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Abstract 

The  advancement  in  small  unmanned  aerial  vehicle  (SUAV)  technology  has 
brought  a  new  revolution  in  the  military  domain.  Their  uses  have  become  more 
synonymous  with  intelligence,  surveillance,  and  reconnaissance  missions.  Concerns  over 
their  flight  test  safety  and  accountability  have  been  addressed  in  multiple  policies  to 
mitigate  mishaps  and  increase  proper  accountability.  However,  current  DoD  and  FAA 
mandated  regulations  and  policies  concerning  UAV  flight  tests  are  sometimes  considered 
slow  and  time-consuming,  which  may  lead  to  delays  in  UAV  research  and  development. 

This  study  explores  the  quantitative  and  qualitative  measure  of  benefits  associated 
with  an  abbreviated  flight  test  process  for  SUAVs.  Specifically,  it  examines  the  current 
agreement  between  two  major  USAF  research  centers  regarding  the  SUAV  flight  test 
approval  process.  This  research  utilized  high-level  multidisciplinary  approaches  and 
techniques  including  qualitative  cost-benefit  analysis,  interviews,  value  stream  mapping 
(VSM)  analysis,  and  heuristic  risk  analysis  to  evaluate  the  current-state  process.  The 
findings  conclude  that  there  is  a  slight  economic  cost  and  schedule  savings  in  an 
abbreviated  process.  Additionally,  this  research  finds  no  correlation  between  SUAV 
flight  mishaps  and  system  maturity.  This  research  proposes  using  a  streamlined  process 
for  additional  safety  reviews  to  eliminate  non- value  added  process  steps  considered 
unnecessary  due  to  the  nature  of  the  SUAV  complexity.  Furthermore,  this  study 
recommends  using  a  decision  rule  matrix  based  on  the  total  cost  of  the  SUAV  and  its 
weight  and  energy  at  impact  for  choosing  an  abbreviated  flight  test  safety  review  process. 
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UNMANNED  AERIAL  VEHICLE  FLIGHT  TEST  APPROVAL  PROCESS  AND  ITS 
IMPLICATIONS:  A  METHODOLOGIAL  APPROACH  TO  CAPTURE  AND 
EVALUATE  HIDDEN  COSTS  AND  VALUE  IN  THE  OVERALL  PROCESS 

I.  Introduction 

“For  once  you  have  tasted  flight,  you  will  walk  the  earth  with  your  eyes  tuned  skywards, 
for  there  you  have  been  and  there  you  will  long  to  return.  ”  -  Leonardo  da  Vinci 

Historically,  humans  have  approached  flight  test  process  and  evaluation  with 
scientific  methods  since  the  early  days  of  aviation.  Mankind  has  always  been  fascinated 
with  flight  and  the  ability  to  control  the  sky.  Examples  range  from  early  attempts  such  as 
Greek  mythology’s  Icarus’  wings  to  real  scientific  approaches  like  Leonardo  da  Vinci’s 
omithopter.  By  the  late  19th  century,  the  race  was  on  to  build  the  world’s  first  powered 
and  controlled  flying  machine.  Numerous  flight  designs  and  tests  were  done  by  scientists, 
engineers,  and  flight  enthusiasts  competing  for  the  title  of  the  first  to  sustain  heavier- 
than-air  human  flight;  and  in  the  process,  all  of  them  took  some  form  of  risks,  including 
those  to  their  own  safety.  In  1903,  brothers  Orville  and  Wilbur  Wright  took  their  first 
successful  human-controlled  airplane  to  the  air  in  that  historic  twelve-second  flight  and 
created  a  new  chapter  in  human  innovation.  There  is  no  doubt  that  the  Wright  brothers 
had  tested  various  methods  for  their  conceptual  flight  and  preliminary  designs  before  they 
could  control  the  craft  with  three-axis  control  and  the  correct  airfoil  design.  These 
pioneers  of  aviation  who  experimented  with  gliders  and  biplanes  laid  the  groundwork  for 
later  generations  to  build  more  sophisticated  advancements  in  aerodynamics  and  engine 
technology  that  will  power  them  to  new  frontiers. 
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Fast  forward  a  hundred  years  later,  human  ingenuity  and  breakthroughs  in 
maehinery  have  enabled  mankind  to  travel  to  spaee  and  control  the  air  from  the  ground  a 
thousand  miles  away.  The  risks  and  safety  involved  in  experimenting  with  new 
technology  have  always  been  a  challenge  to  those  who  flight  tested  them;  those  who  tried 
to  experiment  with  aerial  control  will  often  find  themselves  confronted  with  failures  that 
may  result  in  putting  their  lives  in  harm’s  way.  Despite  the  risks  associated  with  flight 
tests,  many  organizations  and  government  centers  continue  to  set  improvement  targets 
with  different  purposes  to  control  the  sky  using  various  methods  including  the  newest 
marvel  of  unmanned  aerial  vehicles  (UAV). 

1.1  USAF  Flight  Test  Process 

Different  sources  and  organizations  reference  their  UAVs  in  different  ways.  The 
nomenclature  for  an  unmanned  aerial  vehicle  is  not  standard  across  the  industry;  some 
have  termed  these  vehicles  as  remotely  piloted  vehicles  (RPV),  or  remotely  operated 
aircraft  (ROA),  or  unmanned  aerial  systems  (UAS).  For  the  purpose  of  this  paper,  the 
term  unmanned  aerial  vehicle  (UAV)  will  be  used  to  indicate  any  aircraft  capable  of 
flight  without  a  human  operator  on  board.  Small  UAVs  (SUAV)  specify  vehicles  that  are 
measured  a  few  feet  across  in  length  and  wingspan,  and  these  are  typically  found  in 
surveillance,  urban  terrain  operation,  communications  relay,  and  reconnaissance. 

Recent  development  and  the  ease  of  access  to  global  positioning  systems  (GPS) 
and  remote  controlled  instruments  have  led  to  an  increase  in  SUAV  research  and 
development  (R&D)  in  the  United  States  (U.S.)  and  the  world  in  general.  It  is  now 
possible  for  a  radio-controlled  aircraft  builder  to  manufacture  SUAVs  based  on  user 
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requirements  and  needs.  The  acceptable  low  risk  and  low  cost  in  SUAV  systems  to 
operate  in  the  battlefield  have  gained  the  attention  of  the  military,  causing  it  to  engage  in 
more  R&D.  Acceptable  low  risks  and  low  costs  are  generally  defined  as  the  likelihood  of 
an  event  whose  probability  of  occurrence  is  subjected  to  an  acceptable  risk  at  a  minimal 
loss  but  perceived  as  having  greater  benefits.  The  rapid  pace  of  SUAV  manufacturing 
processes  has  enabled  many  organizations  and  the  military  to  conduct  a  different  means 
of  research  that  may  prove  critical  to  national  defense  and  security  by  utilizing  SUAVs  to 
perform  various  missions,  such  as  border  patrol;  intelligence,  surveillance,  and 
reconnaissance  (ISR);  and  autonomous  surveillance  and  patrol  tasks  on  the  battlefield. 
Presently,  flight  approval  processes  for  SUAVs  in  the  U.S.  national  airspace  are 
mandatory  in  order  to  satisfy  flight  requirements  as  regulated  by  the  Federal  Aviation 
Administration  (FAA).  Routinely,  this  process  can  sometimes  take  a  few  weeks  to  a  few 
months  (FAA,  2011). 

There  is  an  abundance  of  literature  that  describes  the  flight  test  approval  process 
for  larger  and  manned  aircraft,  including  those  from  the  U.S.  Air  Force,  U.S.  Navy,  FAA, 
and  other  national  test  ranges  (Ward,  2006;  Ingham,  2006;  De  Garmo,  2004).  Similar  to 
manned  aircraft,  the  need  to  increase  safety,  reliability,  and  accountability  on  UAV 
performance  necessitates  flight  test  approval  processes,  as  these  SUAVs  are  becoming 
more  autonomous.  The  purpose  of  a  flight  test  approval  process  is  to  bring  in  multiple 
experts  from  different  disciplines  to  strengthen  the  shared  safety  and  technical 
responsibility  between  the  approving  authority  and  the  operators.  It  is  aimed  at  improving 
standards  in  aviation  safety  and  technical  feasibility  of  the  vehicle.  Normally,  a  flight  test 
or  ground  test  approval  process  accomplishes  two  main  objectives:  (1)  finding  and  fixing 
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any  identified  aireraft  design  problems  in  the  test  plan  and  safety  plan  and  (2)  addressing, 
verifying,  and  documenting  the  flight  test  authority  and  the  responsibility  of  the  people 
involved  in  the  process  according  to  applicable  standards  and  regulations  (Ward  et  al., 
2006).  These  two  main  objectives  bring  the  flight  test  process  to  a  minimal  level  of 
acceptable  risk,  which  is  the  lowest  risk  possible  after  all  hazards  have  been  eliminated  or 
mitigated  (Ward  et  al.,  2006). 

1.2  Problem  Statement 

Currently,  numerous  safety  requirements  as  well  as  lessons  learned  from  previous 
tests  have  resulted  in  improved  UAV  design,  engineering,  and  maintenance.  Most  flight 
test  techniques  and  procedures  for  SUAVs  are  rather  simple  due  to  their  designs  and  low 
cost  and  maintenance.  However,  regardless  of  how  simple  or  complex  these  SUAVs  can 
be,  they  are  normally  treated  the  same  as  manned  aircraft  when  it  comes  to  the  flight  test 
approval  process  in  the  USAF,  and  also  by  the  FAA.  There  are  also  challenges  created  by 
flight  mishaps  that  require  additional  time  and  resources  to  approve  the  “return  to  flight” 
of  the  mishap  UAV.  A  return  to  flight  approval  is  required  once  a  vehicle  is  involved  in  a 
crash  or  mishap  that  results  in  property  damage  or  injuries.  The  current  UAV  flight  test 
approval  process  assumes  the  same  resources  and  manpower  are  available  that  the 
majority  of  manned  and  larger  aircraft  projects  have.  Given  the  design  and  developmental 
scope  of  SUAVs  and  their  lack  of  sophisticated  hardware,  it  is  hypothesized  that  the 
predominant  flight  test  approval  process  for  manned  and  larger  aircraft  may  not  be 
suitable  for  SUAVs. 
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To  improve  the  process  for  SUAV  flight  tests,  a  recommendation  is  needed  to 
determine  the  overarching  applicable  standards  and  regulations  for  a  SUAV  flight 
approval  process.  This  new  recommendation  should  sufficiently  increase  the  plarming 
and  development  process  for  SUAVs  that  may  be  targeted  for  rapid  product  development 
for  the  warfighters,  while  adhering  to  safety  and  flight  test  regulations  and  risk 
management. 

1.3  Research  Objectives 

In  narrowing  the  scope  of  this  study,  the  main  objectives  are  to  establish 
quantitative  and  qualitative  measures  of  benefit  for  an  abbreviated  flight  test  process  for 
SUAVs  relative  to  its  resource  and  schedule  costs  versus  using  the  flight  test  process  used 
primarily  for  manned  and  larger  aircraft.  In  particular,  the  research  will  focus  on  the  sizes 
and  costs  of  the  UAV  as  a  way  to  determine  the  appropriate  recommendation  for  a  formal 
flight  approval  process.  Additionally,  this  research  will  also  determine  whether  or  not 
SUAV  flight  tests  should  adhere  to  current  standards  and  regulations  based  on  their 
simplicity,  cost,  and  urgency.  Finally,  this  research  will  explore  the  benefits  and  risks 
involved  when  not  using  the  current  process  as  set  out  by  the  USAF. 

1.4  Assumptions  and  Limitations 

Several  significant  assumptions  were  required  for  the  comparison  of  the  flight  test 
approval  process  between  SUAVs  and  larger  manned  aircraft  to  be  feasible  and  practical. 
One  primary  assumption  is  the  work  value  in  resources  and  manpower  added  to  each 
program  will  be  different  from  each  other;  particularly,  SUAV  programs  do  not  require  as 
many  resources  and  manpower  as  those  of  larger  manned  aircraft  programs.  Another 
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assumption  is  the  system  complexity  and  safety  guidelines  of  both  types  of  vehicles  are 
assumed  to  be  different  due  to  the  nature  of  manned  versus  unmanned  aircraft  (i.e., 
manned  vehicles  are  more  complex  and  require  a  higher  standard  of  safety  guidelines 
than  unmanned  SUAVs).  Lastly,  the  risk  assessment  and  management  from  both  classes 
of  vehicles  are  also  assumed  to  be  different,  that  is,  SUAVs  are  sometimes  expendable 
because  of  their  low  manufacturing  and  operating  costs. 

There  are  more  than  a  few  flight  test  processes  nationwide  (Ward  et  ah,  2006), 
each  with  different  approving  authority  and  policies  to  make  flight  tests  happen.  Since 
this  is  exploratory  research,  this  research  will  use  the  flight  test  process  being  utilized  at 
Edwards  Air  Force  Base  (AFB)  to  make  assessments  and  recommendations  on  the  flight 
test  approval  process  and  its  implications  associated  with  SUAVs.  It  is  recommended  that 
future  follow-on  research  should  seek  to  validate  and  generalize  this  approach  and 
methodology. 

1.5  Thesis  Overview 

Chapter  2:  Literature  Review,  will  examine  the  background  in  flight  test  approval 
process  and  its  implementation,  as  well  as  the  literature  on  the  current  flight  test  process 
and  issues  with  traditional  flight  test  methods  for  SUAVs.  Chapter  3:  Methodology,  will 
discuss  the  methodology  in  collecting  data  and  how  each  step  in  the  process  will  be 
analyzed,  as  well  as  the  scope  of  the  projects  involved.  Chapter  4:  Results  and  Analysis, 
provides  analysis  and  results  of  this  research.  And  finally.  Chapter  5:  Discussions  and 
Conclusions,  will  summarize  this  research  and  present  findings  and  recommendations  for 
the  flight  testing  SUAVs,  along  with  recommendations  for  future  research. 
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II.  Literature  Review 


The  purpose  of  this  chapter  is  to  understand  the  current  literature  on  the  small 
unmanned  aerial  vehicle  (SUAV)  flight  test  approval  processes.  This  chapter  starts  with 
the  history  of  flight  test  requirements  and  reviews  the  Memorandum  of  Agreement 
(MO A)  between  the  Air  Force  Research  Laboratory  (AFRL)  and  the  Air  Force  Flight 
Test  Center  (AFFTC)  covering  flight  operations  oversight  of  AFRL  flight  test  programs 
and  the  authority  and  responsibilities  from  both  organizations.  The  importance  of  flight 
tests  will  be  discussed  with  factors  affecting  flight  test  methods.  A  short  overview  of  the 
large  potential  military  market  for  SUAVs  will  be  presented.  Lastly,  a  brief  description  of 
various  popular  known  flight  certification  processes,  including  those  from  the  FAA  and 
other  international  agencies,  will  be  discussed. 

2.1  MOA  between  AFRL  and  AFFTC 

Before  AFFTC  involvement  in  AFRL’s  flight  test  process,  AFRL  had  been 
conducting  a  series  of  manned  and  unmanned  flight  tests  ranging  in  various  sizes  from 
desk-sized  UAVs  to  larger  manned  cargo  aircraft.  However,  after  a  series  mishaps  and 
accidents  (AFRL  T&E  Office,  201 1),  the  Air  Force  Materiel  Command  (AFMC)  directed 
that  a  rated  Operations  Group  Commander  provide  oversight  for  all  flying  activities 
involving  military,  government  civilian,  or  contractor  flight  operations  (AFMC/A3, 

2006).  By  AFMC  direction,  the  412*  Operations  Group  (OG)  was  identified  as  the  Flight 
Operation  Authority  (FOA)  for  all  AFRL  flight  test  programs,  except  for  the  AFRL 
Munitions  Directorate  (AFRL/RW)  for  which  the  46*  OG  (Eglin  AFB)  provides  this 
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oversight.  Edwards  AFB  is  the  home  of  AFFTC,  including  the  412*  OG,  and  is  one  of 
the  major  flight  test  centers  in  the  U.S.  with  the  expertise  to  support  and  provide  guidance 
to  program  managers  (PM)  at  AFRF  before  approving  any  flight  operations.  The 
requirement  for  the  412*  OG  involvement  in  AFRF  Test  and  Evaluation  (T&E)  is 
formalized  in  the  MOA  that  was  established  in  May  of  2006  (2006-06-06MA).  The  MOA 
states: 


“The  AFFTC  has  responsibility  for  the  safe  and  effective  conduct  of flight 
operations  for  AFRF  research  programs  requiring  flight  activities.  The 
AFMC/A3  defers  OPCON  to  the  412^^  Operations  Group  Commander 
(OG/CC)  for  AFRF  research  programs  involving  military,  government 
civilian  or  contractor  flight  operations.  To  avoid  duplicate  reviews, 
inspections  or  evaluations,  an  integrated  team  approach  for  conducting 
program  flight  operation  oversight  will  be  used.  " 


Deferring  operational  control  (OPCON)  to  AFFTC  for  all  AFRF  flight  operations 
includes  a  variety  of  programs  ranging  from  small  remote  controlled  (RC)  aircraft,  such 
as  micro  sensor  test-beds,  to  full-scaled  aircraft  with  advanced  composite  structures.  The 
intent  of  the  MOA  is  to  have  experienced  flight  testers  review  AFRF  programs  by 
providing  insight  and  involvement  on  flight  test  programs  by  test  pilots  and  flight  test 
engineers.  The  personnel  who  review  AFRF  test  plans  are  usually  those  who  have 
experience  with  flight  operations  and  also  those  who  have  graduated  from  the  USAF  Test 
Pilot  School,  which  is  also  at  Edwards  AFB. 

There  are  certain  requirements  and  procedures  that  AFRF  and  AFFTC  must 
collaborate  together  on  to  have  a  flight  test  program  approved  for  flight  status.  These 
responsibilities  include  AFRF  providing  monthly  summaries  for  flight  activities  and 
notes,  as  well  as  UAV  airworthiness  from  their  safety  review  boards  (SRB).  Furthermore, 
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AFRL  conducts  technical  review  boards  (TRB)  and  SRB  in  addition  to  doeumenting  all 
flight  authorizations  from  AFRL  and  AFFTC.  AFFTC  responsibilities  include  reviewing 
and  approving  test  plans  and  safety  plans  in  accordanee  with  Federal  Aviation 
Regulations  (FAR),  Air  Foree  Instructions  (AFI),  and  AFMC  and  AFFTC  supplements. 

In  the  ease  of  a  flight  mishap,  AFFTC  may  provide  assistance  to  the  return  to  flight  board 
and  assist  in  technical  and  safety  reviews  (AFMC/ A3,  2006).  More  information  on  the 
MOA  between  AFRL  and  AFFTC  can  be  found  in  Appendix  A. 

Normally,  a  flight  test  proeess  begins  at  the  AFRL  T&E  Offiee,  as  shown  in 
Figure  2.1.  The  AFRL  T&E  Office  accepts  a  flight  test  program  from  one  of  its 
direetorates  and  begins  coordinating  within  AFRL.  Following  AFRL  SRBs  and  TRBs, 
the  test  paekage  is  submitted  to  the  412*  OG  for  review.  In  almost  all  cases,  the  412*  OG 
provides  eomments  and  questions  eonceming  flight  safety  and  operational 
responsibilities  and  polieies  back  to  AFRL  T&E  Office  for  action.  The  review  package 
can  include  a  test  plan,  safety  plan,  crew  qualifieations,  results  from  AFRL  SBR,  and 
flight  authorizations.  Comments  and  review  procedures  are  usually  handled  over  the 
phone  or  through  email  due  to  the  two  geographieally  separated  loeations.  This  overall 
proeess  may  take  from  three  to  six  weeks  for  coordination  to  complete.  Flight  programs 
that  are  assessed  as  medium  to  high  risks  may  take  longer  due  to  higher  safety  and 
technical  requirements  addressed  by  FAR,  AFI,  and  AFFTC  regulations. 
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Figure  2.1:  AFRL  and  AFFTC  Flight  Test  Approval  Process 
(445*  Flight  Test  Squadron,  2008) 


In  addition  to  safety  reviews  from  both  AFRL  and  AFFTC,  the  PM  is  required  to 
submit  a  vehicle  airworthiness  application  for  flight  operations  in  accordance  with  AFI 
62-6  if  the  vehicle  is  to  be  flown  outside  restricted  airspace.  The  airworthiness 
determination  is  independent  from  the  PM’s  chain  of  execution.  The  Technical 
Airworthiness  Authority  (TAA)  is  the  authority  to  make  these  determinations  for  safe 
flight  operations  and  approve  all  certification  basis  documents  and  issue  Military  Type 
Certificates  (MTC)  that  prove  the  aircraft  system  type  is  in  full  compliance  with  the  FAR. 


2.1.1  Risk  Assessment  and  Test  Approval  Authority 

As  determined  by  the  MO  A  presented  earlier,  the  Test  Approving  Authority 
(TAA)  from  AFRL  approves  AFRL  flight  test  programs  while  AFFTC  approves  the 
AFRL  programs  through  the  FOA.  The  AFI  and  Air  Force  Flight  Test  Center  Instructions 
(AFFTCl)  for  flight  tests  mandate  a  higher  authority  review  and  approval  for  any 
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increase  in  risks.  Per  AFFTCI  91-5  and  AFI  99-103,  the  signature  authorities  for  low, 
medium,  and  high  risk  test  plans  are  detailed  in  Table  2.1. 


Table  2.1:  AFFTC  Signature  Authority 
(AFFTCI  91-5  AFFTC  Test  Safety  Review  Proeess) 


ORGANIZATIONAL 

LEVEL 

LOW  RISK 

MEDIUM 

RISK 

HIGH  RISK 

Operations  Group 
Commander  (OG/CC) 
or  equivalent 

Approve 

Coordinate 

Coordinate 

Test  Wing  Commander 
(TW/CC)  or  equivalent 

Information 

Approve 

Coordinate 

AFFTC  Commander 
(AFFTC/CC) 

Information 

Information 

Approve 

Low  risk  test  plans  are  approved  by  the  412*  OG  Commander  (412  OG/CC)  and 
the  owning  AFRL  Division  Chief  or  their  deputies  before  the  start  of  any  flight 
operations.  Medium  risk  test  plans  are  jointly  approved  by  the  412*  Test  Wing 
Commander  (412  TW/CC)  and  the  owning  AFRL  Teehnieal  Directorate  Director.  High 
risk  test  plans  are  jointly  approved  by  the  AFFTC  Commander  (AFFTC/CC)  and  the 
AFRL  Commander  (AFRL/CC).  All  coordination  comments  between  AFFTC  and  AFRL 
must  be  resolved  before  a  final  approval  for  test  execution.  To  avoid  any  duplicate 
reviews,  inspeetions,  or  evaluations  by  both  organizations,  an  integrated  test  team 
approach  is  used  for  condueting  flight  operations  oversight. 


2.1.2  Performance  Gaps 

Although  instruetions  were  provided  by  the  AFRL  T&E  Office  PMs  on 
proeedural  methods  to  conduet  flight  tests  and  flight  approval  process,  most  of  the  test 
plans  and  safety  plans  are  deemed  inadequate  by  the  AFRL  T&E  Office,  whieh  ean  be 
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traced  to  the  laek  of  flight  test  experienee  and  details  in  the  preparation  of  test  plans. 
Additional  problems  are  caused  because  the  majority  of  the  PMs  never  sent  initial 
notifications  of  pending  tests  to  the  AFRL  T&E  office.  In  2010,  over  40  test  plans  were 
submitted  from  different  directorates  to  the  AFRL  T&E  office,  90%  of  which  were 
deemed  low  risk  status,  but  over  50%  of  the  test  plans  submitted  for  coordination  were 
only  given  a  three- week  notice  in  a  proeess  that  normally  requires  six  months  of  lead 
time  (AFRL  T&E  Office,  201 1). 

Previous  researeh  efforts  condueted  by  Homan,  Parker,  Wirthlin,  and  Pignatiello 
(201 1)  found  that  the  number  of  test  plans  submitted  to  AFFTC  ereates  a  potential  major 
work  bottleneck  at  the  FOA  at  Edwards  AFB,  who  is  also  responsible  for  nine  other  local 
test  organizations.  Moreover,  these  loeal  flight  tests  are  the  FOA’s  main  duties  at 
Edwards  AFB,  and  they  do  usually  take  preeedence  over  approving  AFRL  flight  test 
programs.  Another  major  problem  is  the  workload  and  how  it  flows  through  the  AFRL 
approval  process,  thus  further  delaying  the  flow  of  the  test  plans.  As  seen  in  Figure  2.2, 
members  at  the  AFRL  T&E  Office  handle  numerous  test  plans  per  year  submitted  from 
dozens  of  PMs  from  different  AFRL  directorates.  These  test  plans  are  sometimes  written 
in  non-standardized  formats  and  are  usually  submitted  late.  This  tardiness  in  most  of  the 
test  plans  delays  the  AFRL  T&E  Office  reviewing  the  test  plans  and  hampers 
coordinating  with  the  members  at  AFFTC. 
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Figure  2.2:  AFRL  and  AFFTC  Flight  Approval  Performance  Analysis 
(Modified  from  Homan  et  al.,  201 1) 


In  the  case  of  a  mishap,  the  retum-to-flight  process  also  uses  the  same  test 
approval  process  and  is  usually  handled  by  the  same  members  approving  the  original  test 
plans.  Upon  learning  of  a  mishap,  the  AFRL  T&E  Office  and  the  test  team  are  required  to 
notify  AFFTC  of  the  mishap.  To  mitigate  further  risks,  the  test  team  is  required  to  stop  all 
tests  and  file  a  report  on  the  mishap,  which  takes  more  time  and  effort  away  from  any 
other  on-going  coordination  with  the  AFRL  T&E  Office  and  AFFTC.  The  reporting 
process  begins  when  the  mishap  investigation  is  initiated  by  an  integrated  test  team  from 
both  AFRL  and  the  test  team.  AFRL  holds  all  accountability  and  investigation 
responsibilities  during  the  mishap.  The  412*  OG  is  the  ground  and  retum-to-fly  approval 
authority  for  mishaps  that  are  considered  Class  A  or  Class  B,  which  include  permanent, 
partial  or  total  disability  or  property  damage  of  $500,000  or  more  (AFMC/A3,  2006, 
AFRLM  99-103). 
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2.1.3  Improvement  Initiatives 

The  performance  gaps  and  issues  presented  earlier  reveal  a  need  for  change  in  the 
process,  manpower,  and  resources  for  management  and  improvement.  Collectively,  there 
is  room  for  improvement  on  both  sides  of  the  process.  Any  new  PM  to  the  AFRL  flight 
test  process  would  notice  the  work  burden  carried  by  the  AFRL  T&E  Office;  yet,  there  is 
still  another  side  to  the  issue,  which  is  the  work  of  approval  authority  at  AFFTC.  The 
FOA  at  AFFTC  is  often  times  regarded  as  the  single  chokepoint  and  sometimes  blamed 
for  the  failure  to  deliver  on-time  approval  of  flight  dates  (Homan  et  ah,  2011).  Although 
AFFTC’s  main  concern  for  AFRL  flight  test  programs  is  the  safe  conduct  of  flight  tests 
in  accordance  with  applicable  AFFTC,  FAA,  and  USAF  instructions  and  their 
supplements,  their  main  duty  lies  with  nine  other  local  flight  test  organizations  or 
squadrons  (Homan  et  ah,  201 1).  Therefore,  any  quick  reviews  from  AFFTC  may 
jeopardize  the  safety  of  the  program. 

Many  discussions  for  improvements  and  exchange  initiatives  have  been  raised  by 
both  sides  to  speed  up  the  process,  but  few  have  come  to  full  fruition  (AFRL  T&E  Office, 
2011).  One  of  the  initiatives  that  started  from  both  sides  of  the  process  was  an  agreement 
to  draft  an  amendment  to  the  original  MOA  to  eliminate  AFFTC  oversight  on  low  risk 
programs  and  allow  AFRL  the  authority  to  approve  its  own  flight  test  programs  without 
frill  AFFTC  involvement  (AFMC/A3,  2006).  This  initiative  was  frinneled  up  to 
AFMC/A3  asking  for  AFRL  to  assume  full  approval  authority  and  responsibility  for  all 
low  risk  programs,  with  the  412*  OG  continuing  to  provide  oversight  for  medium  and 
high  risk  programs  as  defined  in  the  MOA.  However,  the  initiative  was  quickly  rejected 
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at  AFMC/A3  after  weighing  the  risks  and  benefits  of  having  no  Operations  Group 
Commander  oversight  of  an  AFRL  flight  test  program  (HQ  AFMC/A3F). 

2.2  The  Importance  of  Flight  Tests 

Flight  testing  in  both  piloted  and  unmanned  aerospace  vehicles  is  an  ongoing 
activity  that  pushes  the  limits  of  aeronautical  and  aerospace  knowledge.  The  primary 
purpose  of  flight  testing,  according  to  the  AGARD  Flight  Test  Manual  (Perkins  et  al., 
1959),  lists  three  fundamental  reasons  for  flight  testing: 

1.  To  determine  the  actual  characteristics  of  the  machine. 

2.  To  provide  developmental  information. 

3.  To  obtain  research  information. 

Although  the  reasons  listed  above  do  not  specifically  indicate  that  modem 
aerospace  vehicles  require  teams  of  engineers  and  scientists,  whether  piloted  or 
unmanned;  they  do,  however,  provide  the  classical  reasons  for  conducing  flight  tests. 
From  this  perspective,  the  trae  purpose  of  flight  testing  is  to  reduce  future  uncertainties  in 
the  design  and  test  processes  before  going  to  the  operational  stages  and  airworthiness 
certifications.  Flight  testing  also  provides  practical  experiences  to  the  operators  and 
involves  interaction  with  other  systems  that  increases  interoperability  (Williams  & 

Harris,  2002). 

Recent  advances  in  technologies  and  new  procedures  may  have  been  successfully 
integrated  into  the  stages  of  aerospace  development,  but  basic  flight  tests  are  required  to 
provide  credibility,  and  these  must  be  proven  in  the  flight  test  environment  including 
piloted  aircraft  and  UAVs.  Additionally,  the  involvement  of  humans  in  UAV  missions  in 
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flight  tests  ensures  that  the  total  UAV  system  functions  correctly.  Flight  tests  and 
evaluations  must  also  determine  the  suitability  of  the  UAV  system  for  use  by  the  end 
users,  especially  the  operators  and  maintainers  (Williams  &  Harris,  2002). 

Historically,  military  systems  have  relied  heavily  on  prototypes  of  airplanes  to 
provide  valuable  predictions  for  development  efforts.  Since  the  early  1970s,  the  YF-22 
and  YF-23,  the  YF-16  and  YF-17,  the  YA-9  and  YA-10,  and  the  X-32  and  X-35  efforts 
were  all  head-to-head  flight  test  comparisons  of  competing  prototypes  (Ward  et  al., 

2006).  Prototyping  does  not  always  lead  to  production,  but  may  instead  lead  to  further 
development  of  an  existing  design.  Subsystems  technology  must  also  be  proven  in  the 
harsh  flight  environment  (Ward  et  al.,  2006).  These  subsystems  include  radar, 
navigational  systems,  flight  management  systems,  integrated  displays,  flight  control 
system,  communication  systems,  etc. 

According  to  the  UAV  Task  Force  Final  Report  (2004),  unlike  manned  aircraft 
which  are  integrated  by  a  network  communication  with  air  traffic  control  (ATC),  flight 
management,  and  flight  operation  from  various  flight  control  operators,  UAVs  are 
dependent  on  a  network  of  remote  systems  and  subsystems  to  make  flight  operations 
possible.  It  is  for  this  reason  that  testing  and  evaluation  needs  to  be  executed  on  the  entire 
system  of  the  UAV  and  not  on  the  vehicle  alone.  This  could  raise  the  magnitude  of 
testing  a  relatively  uncomplicated  SUAV  to  that  of  testing  a  highly  complex  integrated 
system  (Williams  &  Harris,  2002). 
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2.3  Factors  in  Flight  Testing 

It  takes  thousands  of  man-hours  and  resources  to  build  a  new  aircraft,  but  to  get 
certification  to  be  airworthy  for  flights,  every  part  of  an  aircraft  has  to  meet  standards  set 
out  by  the  military  and  certifications  required  from  the  FAA.  Testing  a  flight  vehicle  is  a 
complex  process.  It  requires  careful  planning  and  measures  of  completeness  and  accuracy 
in  the  data  to  be  safe,  within  budget,  and  on  time.  In  any  form  of  tests,  the  most 
significant  cost  for  any  program  is  the  cost  of  safety.  Therefore,  almost  all  policies  or 
redundancies  address  safety  concerns  throughout  the  planning  and  conduct  of  any  series 
of  flight  tests.  Ward,  et  al.  (2006)  discus  the  three  primary  factors  in  the  test  planning 
process  as  the  dominant  concerns  for  all  PMs  and  testers  alike. 

2.3.1  Safety 

Every  test  carries  some  element  of  risk,  whether  it  is  a  low,  medium,  or  high  risk 
of  a  new  prototype  or  determining  the  performance  of  a  new  piece  of  software  on  an 
aircraft  system.  Risk  elimination  is  a  responsibility  carried  by  all  who  are  involved  in 
flight  testing.  Almost  all  flight  test  centers  endorse  a  checklist  approach  toward  safety 
concerns  based  on  the  culmination  of  past  experience  and  what  seems  to  be  the  best 
methodology  in  risk  management.  There  is  no  single  way  to  manage  risk  in  engineering 
processes;  however,  many  flight  test  organizations  (both  government  and  industry)  use  a 
standard  set  of  procedures  common  to  flight  tests  to  manage  risks.  The  following 
summary  is  based  on  the  AFFTC  and  NASA  Procedures  and  Guidelines  on  T&E  control 
and  conduct  of  risk  assessment  and  risk  mitigation  (Ward  et  al.,  2006).  Most  of  the  risk 
management  process  includes  the  following  steps: 
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1.  Hazard  definition 

2.  Cause  identification  /  Risk  assessment 

3.  Risk  mitigation 

4.  Emergency  response 


Test  specific  hazards  are  those  that  arise  as  a  direct  consequence  of  the  test 
activity.  This  step  explores  some  of  the  “what  if’  questions  that  might  increase  the 
probability  of  the  hazard  event  occurring  (Ward  et  ah,  2006).  Cause  identification  lists 
the  likelihood  of  occurrence  ranging  from  improbable  to  frequent  that  the  cause  should 
occur  at  least  once.  The  severity  of  consequence  of  the  cause  ranges  from  negligible  to 
catastrophic,  which  includes  injury  and  death.  Table  2.2  shows  an  example  of  a  matrix  of 
the  hazard  probability  and  its  severity  category. 


Table  2.2:  Example  Risk  Assessment  Matrix 
(AFFTCI  91-5  AFFTC  Test  Safety  Review  Process) 


HAZARD  SE\  ERITY  CATEGORY 

Catosti'ophic 

Critical 

Marginal 

Ne^Ii^ible 

HAZARD 

PROBABILITY 

May  cause  death, 
system  loss 

May  cause  severe 
injuiy.  severe  oc¬ 
cupational  illness, 
or  major  system/ 
facility  damage 

May  cause  minor  injuiy. 
uiuior  occupational 
ilhiess.  or  niuior 
system/facility  damage 

May  cause  less  than 
niuior  iiijiny 
occupational  ilhiess 
or  system/facility 
damage 

FREQUENT 

Likely  to  occiu'  fiequently 
during  the  test 

1 

3 

7 

13 

PROBABLE 

Will  occur  several  times 
during  the  test 

2 

5 

9 

16 

OCCASIONAL 

Likely  to  occiu  sometmie 
dining  the  test 

4 

6 

11 

18 

REMOTE 

Unlikely,  but  possible  to 
occiu  during  the  test 

8 

10 

14 

19 

IMPROB.4BLE 

Highly  unlikely  to  occin 
dining  the  test 

12 

15 

17 

20 

18 


Risk  mitigation  is  the  process  by  which  risks  are  reduced,  moving  them  down  and 
to  the  right  in  the  risk  assessment  matrix  (Ward  et  ak,  2006).  According  to  Ward  et  al. 
(2006),  reduction  in  the  severity  level  usually  requires  specific  hardware  mitigations, 
which  are  modifications  to  hardware  to  enhance  the  test  safety;  other  reductions  in  the 
severity  level  are  procedural  mitigations,  which  are  intended  to  decrease  the  risk  and  the 
likelihood  of  occurrence.  Finally,  every  hazard  analysis  should  include  a  procedural 
response  or  corrective  actions  and  some  planning  to  address  a  mishap  should  it  occur. 

2.3.2  Cost 

The  cost  of  flight  testing  can  range  from  tens  of  thousands  to  millions  of  dollars 
depending  on  the  scale  and  scope  of  each  test.  It  is  therefore  up  to  the  engineers  and  PMs 
to  consider  the  cost  of  each  test  to  be  conducted  and  analyzed  whether  the  test  is 
manageable,  and  most  importantly,  why  the  test  should  be  done.  While  this  research  does 
not  specifically  focus  on  the  factors  of  cost  and  the  efficiencies  of  cost  control  and 
visibility,  it  does,  however,  weigh  the  cost  benefit  and  risk  of  not  having  SUAVs  go 
through  the  formal  flight  test  approval  process  that  larger  manned  aircraft  do  in  the 
USAF. 

Support  for  flight  tests  should  be  kept  as  simple  as  possible  and  consistent  with 
the  risk  and  complexity  of  the  test.  Sometimes,  flight  test  support  can  be  as  simple  as  a 
flight  mechanic  on  the  ground  servicing  the  aircraft  and  as  complex  as  linked  telemetry 
and  continuous  radio  communication  with  all  aircraft  (Ward  et  al.,  2006).  Ignoring  the 
technical  expertise  on  instrumentation  can  lead  to  incorrect  data  and  retest,  which  would 
add  more  cost  to  future  tests.  Finally,  every  second  spent  in  the  air  must  be  used 
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productively.  PMs  should  keep  a  focus  on  combining  series  of  tests  together  as  often  as 
possible  to  reduee  fuel  eosts.  Produetive  flying  time  should  be  used  wisely  but  also 
provide  sufficient  time  to  cheek  for  adequate  data  colleetion  and  test  parameters  (Ward  et 
ah,  2006). 

Costs  may  inerease  if  advaneed  instrumentation  is  involved.  A  SUAV  may  have 
low  manufacturing  cost  and  is  replaceable  if  damaged,  but  the  sensor  payload  attached  to 
the  SUAV  may  eost  even  greater  than  the  SUAV  itself  This  huge  gap  in  the  eosts 
between  the  earrier  and  the  payload  is  one  of  the  challenges  in  the  flight  test  community, 
and  SUAV  regulators  have  yet  to  define  a  definite  risk  analysis  to  the  payload  should  the 
UAV  experience  a  flight  mishap. 

2.3.3  Schedule 

Time  is  money.  Naturally,  nearly  all  flight  test  programs  have  time  and  eosts 
among  their  prime  targets  or  objectives.  One  of  the  biggest  challenges  PMs  must  face  is 
meeting  the  seheduled  deadline.  Flight  eertifieation  and  approvals  usually  take  a  while  to 
get  approved  (DoD  OSD,  2009),  so  it  is  wise  to  plan  ahead  and  schedule  many  months  in 
advance  before  any  actual  test  is  being  done.  A  schedule  slip  can  force  an  entire  program 
to  be  up  for  review  or  face  cancellation.  When  time  and  eosts  are  measured,  performance 
may  be  lost  due  to  eonstraints  in  the  cost  and  schedule.  However,  for  other  projects  such 
as  life  critical  systems,  quality  in  the  performance  would  be  the  overriding  criteria  for 
determining  the  value  of  the  project  (Ward  et  ah,  2006).  A  major  faetor  in  meeting  the 
test  schedule  is  how  many  test  vehieles  are  alloeated  to  the  flight  test  program.  Inereasing 
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the  number  of  flights  or  the  number  of  test  vehicles  may  alleviate  a  schedule  problem,  but 
it  may  not  be  the  answer  when  costs  are  at  a  limit. 


2.4  The  Military  Market 

Recently,  a  market  study  released  by  Teal  Group  Corporation,  an  aerospace  and 
defense  market  analysis  company,  estimates  that  UAV  spending  will  almost  double  over 
the  next  decade  from  current  worldwide  UAV  expenditures  of  $5.9  billion  annually  to 
$12  billion,  of  which  the  U.S.  will  account  for  77%  of  the  worldwide  R&D  and  T&E 
spending  on  technology  and  about  69%  on  procurement  (Teal  Group  Coorporation, 
2009).  Figure  2.3  shows  that  the  U.S.  will  dwarf  the  rest  of  the  world  in  UAV  R&D 
technology  and  procurement  in  the  years  ahead,  followed  by  European  and  Asian-Pacific 
countries. 


World  lAV  Forecast 
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Figure  2.3:  Worldwide  UAV  Forecast 
(Teal  Group  Corporation,  COTS  Journal  Online,  2011) 
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Today,  over  30  countries  are  investing  and  developing  sophisticated  UAVs  for 
multiple  purposes,  including  border  patrol,  global  navigation,  agricultural  spraying,  and 
planting  operations  (De  Garmo,  2004).  The  most  significant  part  of  the  U.S.  R&D  and 
procurement  spending  will  be  for  the  interests  of  the  U.S.  military.  According  to  the 
Center  for  Advanced  Aviation  System  Development  (CAASD),  since  the  1950s,  the  U.S. 
military  has  spent  more  than  $25  billion  on  UAV  development.  However,  it  has  had 
difficulty  in  setting  priorities,  determining  missions,  and  developing  UAV  standards, 
which  has  led  to  a  number  of  programs  being  repeatedly  modified,  replaced,  or  cancelled 
(De  Garmo,  2004).  Despite  the  difficulty  in  setting  UAV  standards,  the  U.S.  DoD  is 
committed  to  setting  the  pace  for  UAV  funding,  research,  and  applications.  In  the  last  ten 
years  alone,  the  U.S.  military  has  increased  over  ten-fold  the  annual  funding  for  UAVs, 
as  shown  in  Figure  2.4.  This  figure  is  expected  to  keep  rising,  eclipsing  robotic  systems 
among  the  DoD’s  top  procurement  priorities  (Sully,  2004). 
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Figure  2.4:  DoD  Annual  Funding  Profiles  for  UAVs 
(De  Garmo,  2004) 


2.5  Regulatory  Requirements  for  SUAVs 

With  a  handheld  GPS  and  a  keen  interest  in  aerodynamics,  hobbyists  and  infantry 
alike  are  becoming  attracted  to  SUAVs.  SUAVs,  normally  powered  by  small  electric  or 
gas  turbine  engines,  are  often  considered  a  subclass  of  UAVs.  These  SUAVs  are 
becoming  increasingly  popular  due  to  their  sizes  and  simplicity  in  data  fielding  and 
operation.  Traditionally,  UAVs  are  primarily  used  by  the  military,  but  demands  for  UAV 
use  have  expanded  to  commercial  and  other  governmental  uses,  which  drives  a 
considerable  need  for  UAV  flight  regulations  and  operational  requirements.  However, 
because  of  the  lack  of  airspace  regulations  and  airworthiness  and  safety  requirements. 
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access  to  UAV  applications  still  present  a  major  barrier  and  ehallenges  to  other  markets 
other  than  the  military  (De  Garmo,  2004). 

An  airworthiness  certifieate  is  an  FAA  doeument  which  grants  authorization  to 
operate  an  aircraft  in  flight  in  compliance  with  the  approved  certifieation  basis  (FAA 
Website,  201 1).  Aeeording  to  AFI  62-601,  airworthiness  is  defined  as  the  ^"verified  and 
documented  capability  of  air  system  configuration  to  safely  attain,  sustain,  and  terminate 
flight  in  accordance  with  approved  usage  and  limits.'’'  A  generie  airworthiness  defines 
whether  an  aircraft  has  been  certified  as  suitable  for  safe  flight  without  significant  hazard 
to  aircrew,  ground  crew,  passengers  (if  relevant),  or  to  the  general  publie  over  whieh  sueh 
airborne  systems  are  flown. 

2.5.1  Issues  with  Traditional  Flight  Test  Methods  for  SUAVs 

Not  all  R&D  technology  and  procurement  come  without  risks  and  mishaps. 
According  to  the  DoD’s  Airspace  Integration  Plan  for  Unmanned  Aviation  (2004), 
SUAVs  have  higher  reported  loss  rates  than  manned  aircraft,  especially  during  takeoff 
and  landing.  Because  of  the  nature  of  UAVs  being  unmanned,  their  loss  eould  sometimes 
be  regarded  as  aeeeptable  in  exehange  for  continuous  data  gathering  and  research  efforts. 

Due  to  the  nature  of  UAVs  and  their  systems,  safety  aspeets  have  yet  to  be 
defined  and  this  leads  to  issues  not  normally  considered  for  manned  systems  (EASA, 
2004).  Aeeording  to  Ostler  (2006),  traditional  flight  testing  methods  may  not  be  suitable 
for  SUAVs  for  a  number  of  reasons.  First,  in-flight  engine  power  cannot  be  determined 
by  measuring  fuel  flow  (i.e.,  eleetrie  engines).  Seeond,  SUAVs  that  are  electrically 
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powered  do  not  decrease  in  weight  over  time  as  do  conventional  aircraft.  Finally,  data 
acquisition  systems  in  SUAVs  are  limited  in  size  and  weight. 

Another  area  usually  not  considered  in  traditional  flight  test  methods  and  the  T&E 
of  SUAVs  is  wind  tunnel  testing  and  aerodynamic  analysis  support.  Although  wind 
tunnel  testing  can  provide  data  to  predict  and  develop  control  surface  designs,  few  SUAV 
operators  opt  to  utilize  wind  tunnel  testing,  as  it  is  generally  accepted  that  flight  testing 
out  in  the  flight  range  is  adequate  enough  and  remains  the  most  reliable  method  for 
SUAVs  (Williams  &  Harris,  2002).  In  addition  to  SUAV  hardware  and  autonomous 
system  reliability,  other  flight  test  challenges  include  categorical  flying  qualities,  flight 
performance,  and  avionics  evaluations.  Provisions  for  these  SUAV  flight  tests  may  not 
completely  follow  conventional  manned  aircraft  flight  test  techniques  due  to  the  nature  of 
UAV  flight  cues  and  their  sensitivities  due  to  of  their  small  physical  sizes  and  weights 
(Williams  &  Harris,  2002). 

Another  area  of  concern  is  the  required  region  of  airspace  possessing  the 
appropriate  vertical  and  lateral  limits  to  conduct  flight  testing  of  UAV  systems  (Williams 
&  Harris,  2002).  This  region  of  airspace  should  have  few  obstructions,  such  as  power¬ 
lines,  tall  structures,  and  trees,  and  it  should  be  located  outside  civil  controlled  airspace. 
Some  SUAVs  are  now  able  to  execute  tasks  autonomously,  such  as  takeoff  and  landing. 
Unfortunately,  the  lack  of  UAV  regulations  has  prevented  these  SUAVs  from  operating 
in  unrestricted  airspace  alongside  manned  aircraft  (Ingham  et  ah,  2006).  Furthermore, 
this  restriction  has  more  complications  due  to  an  unproven  technical  safety  record  where 
the  data  cannot  be  collected  until  more  UAVs  can  operate  alongside  manned  aircraft. 
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which  results  in  a  “chicken  and  egg”  situation  (Ingham,  2006;  Kuke,  2004;  De  Garmo, 
2004). 

2.5.2  UA  V Standards  and  Regulations  in  the  National  Air  Space 

Standards  and  regulations  are  part  of  the  flight  test  process  and  requirements  for 
airworthiness.  By  definition,  a  standard  is  an  accepted  minimum  requirement  in  which 
compliance  must  be  shown  to  achieve  certification  or  qualification  status  (Ingham,  2008). 
UAV  standards  are  currently  under  development  from  the  FAA  for  airworthiness 
certifications.  A  regulation,  such  as  the  FAR,  is  a  set  of  legislative  prescriptions  that  must 
be  complied  with  prior  to,  as  well  as  during,  the  operation  of  a  UAV  in  the  national 
airspace  (Ingham,  2008).  Such  regulations  are  also  under  development  for  civil  UAV 
flights  in  commercial  airspace. 

Although  UAV  markets  for  military  and  civil  applications  have  been  steadily 
growing,  the  lack  of  regulatory  framework  in  creating  standards  and  regulations  has  been 
evident.  With  new  UAV  technology,  major  flight  test  organizations  including  the  FAA 
must  first  determine  the  risks  of  any  associated  system  failures  before  the  new  technology 
is  considered  safe  to  be  put  into  the  National  Airspace  Systems  (NAS)  (Ingham,  2008). 
Any  unforeseen  developments,  technological  improvements,  and  human  factors  play  a 
role  in  whether  the  new  technology  is  safe  enough  to  be  permitted  into  the  NAS.  Once 
the  risks  are  addressed,  flight  safety  and  integration  into  the  NAS  are  taken  in  stages, 
similar  to  the  developmental  testing  and  operational  testing  and  evaluation  (DT&E, 
OT&E)  established  by  the  DoD.  The  FAA  is  also  using  the  same  DT&E  and  OT&E 
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methodology  to  manage  the  integration  of  the  UAV  teehnology  into  the  NAS  (Kalinowki, 

2010). 

Current  recreational  use  of  the  NAS  limits  UAV  operations  to  below  400  feet 
above  ground  level  and  away  from  airports  and  air  traffic  (FAA  Website,  2011).  The 
FAA’s  main  concern  is  not  only  that  UAVs  might  interfere  with  commercial  and  general 
aviation  operations,  but  that  they  could  also  pose  a  safety  problem  for  other  airborne 
vehicles  and  persons  or  property  on  the  ground.  Developing  and  implementing  new  UAV 
standards  and  regulations  is  a  long-term  effort  for  the  FAA,  especially  when  there  is  little 
data  available  to  integrate  UAVs  safely  and  efficiently  into  the  NAS.  The  FAA  also 
indicates  that  more  safety  data  is  needed  before  it  can  establish  a  decision  to  fiilly 
integrate  UAVs  into  the  NAS,  where  the  general  aviation  public  travels  each  day  (FAA 
Factsheet,  2011). 

2.6  Possible  Airworthiness  and  Certification 

The  objective  of  UAV  certification  in  an  unrestricted  airspace  is  to  achieve  “file 
and  fly”  operations,  but  in  order  to  achieve  flight  operations,  flight  tests  will  have  to  be 
executed  in  order  to  prove  compliance  with  relevant  regulations  and  prove  its 
airworthiness  (Ingham,  2008).  Flight  testing  for  airworthiness  in  SUAVs  is  sometimes 
conducted  to  observe  the  usefulness  of  the  latest  developments  for  conventional  aircraft. 
For  example,  a  new  control  law  or  a  novel  wing  structure  was  tested  using  SUAVs  in 
aerial  flight  (Brinker,  2001;  Murray,  2002).  One  of  the  major  challenges  in  operating  an 
SUAV  is  the  capability  to  detect  and  avoid  aircraft  and  other  objects  while  airborne. 
Collision  avoidance  has  become  the  most  pressing  concern,  and  is  also  the  focus  for 
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many  studies  and  research  institutions  worldwide,  including  AFRL  with  its  Auto 
Avoidance  Collision  System  (De  Garmo,  2004). 

To  avoid  collisions  with  other  aircraft,  UAVs  must  have  a  “sense  and  avoid” 
capability,  which  allows  them  to  detect  and  safely  stay  clear  of  other  aircraft  and 
obstructions  midair  (De  Garmo,  2004).  This  development  of  “sense  and  avoid”  capability 
in  UAVs  has  also  provided  significant  benefits  for  both  manned  and  unmanned  operators 
(Ingham  et  al.,  2006).  While  UAV  proponents  view  this  new  technology  as  promising, 
the  limited  safety  and  operational  data  available  does  not  support  the  full  integration  of 
UAVs  into  the  NAS  (FAA  Website,  2011). 

2. 6. 1  Current  Initiatives 

All  UAVs  and  manned  aircraft  in  the  NAS  are  regulated  by  the  FAA  under 
jurisdiction  granted  by  law  (FAA,  201 1).  While  it  is  unarguable  that  large  and  heavy 
UAV  airframes  require  an  airworthiness  certification,  the  question  yet  remains  for 
SUAVs  that  have  wingspans  measuring  less  than  a  few  feet  across  and  weighing  less  than 
a  hundred  pounds.  Do  these  SUAVs  need  the  same  airworthiness  certification  as  those  of 
larger  sized  UAVs?  Some  UAV  enthusiasts  argue  that  if  such  regulations  are  mandated 
on  SUAVs  to  meet  the  same  requirements  for  airworthiness  certification,  it  would 
negatively  affect  airframe  vendors  and  the  SUAV  manufacturing  industry  (UAV 
MarketSpace).  A  recent  safety  study  for  UAV  operation  in  the  NAS  from  MIT’s 
International  Center  for  Air  Transportation  (ICAT)  (2005)  concludes  that  “the  results  of 
the  safety  analysis  indicate  that  it  may  be  possible  to  operate  small  UA  Vs  with  few 
operational  and  size  restrictions  over  the  majority  of  the  United  States.  ” 
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In  response  to  the  influx  of  SUAV  airworthiness  applications,  the  FAA  has 
developed  a  memorandum  titled  “Unmanned  Aircraft  Systems  Operations  in  the  US. 
National  Airspace  System  -  Interim  Operational  Approval  Guidance  ’’  for  guidance  of 
UAVs  in  the  NAS.  The  FAA  is  also  moving  forward  with  UAV  regulations  and 
guidelines  to  minimize  accidents  and  failures.  Furthermore,  the  FAA  is  working  with  the 
Radio  Technical  Commission  for  Aeronautics  (RTCA)  to  develop  technical  UAV 
standards.  Mainly,  RTCA  will  address  two  key  questions  on  UAVs  regarding  their 
handling  of  communication,  command,  and  control,  and  how  UAVs  will  “sense  and 
avoid”  other  aircraft.  According  to  a  statement  by  Kalinowski  before  the  Committee  on 
Homeland  Security  (2010),  these  activities  are  targeted  for  completion  before  2015. 

Other  countries  are  also  publishing  or  have  already  published  regulations  for 
SUAVs.  The  French  Ministry  of  Defense  has  drafted  UAV  regulations  which  are  being 
used  by  other  organizations  as  a  guideline  (Ingham,  2008).  The  Joint 
JAA/EUROCONTROL  Initiative  on  UAVs  final  report  (2004)  proposes  fairness, 
equivalence,  accountability,  and  responsibility  as  guiding  principles  for  integrating  UAVs 
operating  in  civil  airspace.  The  report  emphasizes  the  use  of  existing  manned  regulations 
while  recognizing  the  need  to  tailor  and  complement  them  when  considering  the  specific 
characteristic  of  UAV  operations. 

Other  national  and  international  agencies  are  also  proposing  standards  to  bring 
UAV  flight  operations  within  the  manned  operational  environment  in  conjunction  with 
the  military  and  civil  airspace.  Some  of  these  various  government  and  industry  initiatives 
include:  The  Technical  Analysis  and  Applications  Center  (TAAC),  Joint  Planning  and 
Development  Office  (JPDO)  UA  V  National  Task  Force  (UNTF),  UA  V  National  Industry 
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Team  (UNITE),  American  Society  for  Testing  and  Materials  (ASTM)  UA  V  Committee, 
and  many  other  agencies  worldwide  with  the  objective  of  developing  and  introducing 
UAVs  regulations  in  national  and  international  airspace  (De  Garmo,  2004). 

In  response  to  the  growth  and  development  of  SUAVs,  most  organizations  have 
already  accepted  a  short  cut  by  adopting  manned  aircraft  regulations  that  are  suitable  for 
UAV  flight  operations  (Ingham,  2008;  DoD  OSD,  2004;  Williams  &  Harris,  2002). 
Different  agencies  or  flight  test  organizations  have  different  regulatory  bodies  that  govern 
their  policies  regarding  SUAVs.  Some  agencies  have  proposed  relaxed  regulations 
accordingly  for  academic  institutions,  research  organizations,  and  UAV  developmental 
organizations  that  design  and  test  UAVs  provided  that  these  UAVs  do  not  pose  a  threat  to 
people  and  property  on  the  ground  (Ingham  et  al.,  2006).  For  example,  light  UAVs 
weighing  less  than  150  kg  with  a  kinetic  energy  of  less  than  95  kJ  might  not  be  required 
to  be  certified,  provided  that  they  will  operate  as  radio  controlled  aircraft  within  the 
operator’s  visual  range  in  unrestricted  airspace  for  missions  such  as  telemetry  and  aerial 
photography.  However,  larger  UAVs  weighing  more  than  150  kg  with  a  kinetic  energy  of 
more  than  95  kJ  and  light  UAVs  flying  in  unrestricted  airspace  or  outside  the  visual  range 
of  the  operator  will  be  required  to  be  flight  tested  and  approved  in  a  similar  manner  to 
that  of  manned  aircraft  (CASK  1998,  2001). 

Other  approaches  to  airworthiness  and  certification  of  SUAVs  include  the 
adaptation  of  a  “safety  target”  from  the  European  Aviation  Safety  Agency  (EASA). 

Under  this  approach,  the  overall  objective  of  the  UAV  is  within  the  context  of  a  defined 
mission  and  operating  environment  (EASA,  2004).  The  Safety  Target  approach  is  utilized 
when  UAV  operators  are  under  the  direct  control  of  a  single  authority,  which  has 
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complete  responsibility  for  safety,  and  is  also  the  sole  “eustomer.”  EASA  argues  that 
“this  direct  control  of  operations  is  a  significant  advantage  when  accepting  a  safety  case 
which  relies  upon  the  restriction  of  operations  to  compensate  for  uncertainties  over 
airworthiness  ”  (EASA,  2004). 


2.6.2  DoD  Initiatives 

As  more  UAV  flight  tests  are  being  condueted  by  the  USAF,  military  standards 
have  become  restrietive  regarding  the  safety  and  responsibility  of  the  UAV  programs, 
resulting  in  excessive  costs  in  manpower  and  delays  in  schedule  (Ingham  et  al.,  2006). 
According  to  Ingham,  et  al.  (2006),  these  same  restrietions  have  often  excluded  suitable 
suppliers  because  of  their  non-complianee  with  sueh  speeifieations,  regardless  of  the 
rapid  growth  in  the  number  of  eompanies  with  high  interests  in  the  UAV  business  (Teal 
Group  Corporation,  2011). 

In  1999,  the  DoD,  working  with  the  FAA,  developed  a  flight  approval  proeess 
known  as  a  Certificate  of  Authorization  (CO A)  eontained  in  FAA  Order  7610.4,  Military 
Operations,  that  permitted  UAVs  to  operate  in  the  National  Airspaee  Systems  (NAS)  (De 
Garmo,  2004).  Aceording  to  the  FAA  Faetsheet  (201 1),  the  FAA  issues  a  COA  based  on 
the  following  principles: 


1.  The  COA  authorizes  an  operator  to  use  defined  airspace  and  includes 
special  provisions  unique  to  each  operation.  For  instance,  a  COA  may 
include  a  requirement  to  operate  only  under  Visual  Flight  Rules  (VFR) 
and/or  during  daylight  hours.  Most  CO  As  are  issued  for  a  specified  time 
period  (up  to  one  year,  in  most  cases). 

2.  Most,  if  not  all,  CO  As  require  coordination  with  an  appropriate  air  traffic 
control  facility  and  may  require  the  UAS  to  have  a  transponder  to  operate 
in  certain  types  of  airspace. 
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5.  Due  to  the  UASs  inability  to  comply  with  ’’sense  and  avoid”  rules,  a 
ground  observer  or  an  accompanying  “chase  ”  aircraft  must  maintain 
visual  contact  with  the  UAS  and  serve  as  its  “eyes  ”  when  operating 
outside  of  airspace  that  is  restricted  from  other  users. 

According  to  De  Garmo  (2004)  and  FAA  regulations,  this  process  requires  a  case- 
by-ease  safety  evaluation  of  eaeh  flight,  whieh  results  in  a  lengthy  proeess  that  can  take 
up  to  three  months  to  approve  depending  on  the  FAA  region  or  regions  where  the  UAV 
will  be  flown.  Part  of  the  approval  process  requires  the  FAA  to  issue  a  time  and  route  of 
the  UAV  flight  to  avoid  risks  to  other  aireraft  and  persons  in  the  area,  whieh  further 
limits  the  utility  and  missions  of  UAVs.  The  process  is  incapable  of  sustaining  the  high 
volume  of  UAV  flight  requests  (De  Garmo,  2004).  The  FAA  recognizes  that  the 
proeesses  is  lengthy  and  has  taken  steps  to  streamline  the  process  by  standardizing  the 
review  process  and  inereasing  communieation  and  transpareney  between  the  agency  and 
the  applieants  (Kalinowski,  2010).  Normally,  CO  As  are  worked  on  a  first-eome,  first- 
served  basis,  with  the  exception  of  emergeney  and  disaster  situations  where  the  use  of 
UAVs  warrants  humanitarian  needs. 

2.7  Summary 

This  ehapter  detailed  the  importance  of  flight  tests  and  the  eritical  faetors  in  flight 
testing.  There  is  an  abundanee  of  literature  on  proposals  for  UAV  regulations  and  safety 
standards,  and  this  chapter  briefly  discussed  the  current  initiatives  carried  out  by  the  FAA 
and  other  agencies  in  promoting  UAV  teehnology  in  the  common  airspace.  However, 
additional  UAV  data  and  flight  requirements  are  needed  to  reach  a  decision  for  UAVs  to 
fly  in  the  commereial  airspace.  With  an  understanding  of  the  eomplieations  of  SUAV 
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flight  test  requirements  and  the  background  in  the  MOA  between  AFRL  and  AFFTC,  it  is 
clear  that  additional  studies  are  necessary  to  evaluate  the  process  and  determine  the 
benefits  of  additional  safety  reviews  and  the  costs  associated  with  it.  The  literature  is 
silent  on  this  subject  and  forms  the  basis  of  this  research. 
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III.  Methodology 


This  chapter  reviews  the  methodology  used  in  this  researeh.  The  primary  purpose 
of  the  methodology  is  to  establish  a  quantitative  and  qualitative  measure  of  benefit  for  an 
abbreviated  flight  test  process  for  SUAVs  based  on  the  MO  A  between  AFRL  and 
AFFTC.  This  researeh’ s  methodology  uses  interviews,  cost  sensitivity  analysis,  and 
variability  analysis  using  value  stream  mapping  (VSM)  teehniques  in  the  SUAV  flight 
testing  proeesses  for  the  development  of  a  notional  deeision  rule  matrix.  This  ehapter  also 
discusses  the  development  of  aceeptable  cost  and  optimal  risk  and  the  boundaries  drawn 
from  analysis  to  construet  a  decision  rule. 

3.1  Research  Design 

This  researeh  is  exploratory  in  nature  using  multidiseiplinary  approaches  and 
techniques,  both  quantitative  and  qualitative.  Quantitative  researeh  describes  phenomena 
in  numbers  and  measures  instead  of  words;  the  focus  of  the  researeh  is  usually 
predetermined  and  deducted  from  prior  researeh  (Krathwohl,  1998).  Qualitative  research, 
on  the  other  hand,  describes  phenomena  in  words  instead  of  numbers  or  measures  and 
usually  uses  induction  to  ascertain  what  is  important  in  the  phenomena  (Krathwohl, 

1998). 

By  leveraging  these  multidiseiplinary  teehniques  (qualitative  cost-benefit 
analysis,  interviews,  eost  sensitivity,  variability  analysis,  and  heuristie  risk  analysis),  this 
researeh  will  evaluate  and  quantify  the  current  proeess  used  in  the  AFFTC  review  process 
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and  create  a  decision  rule  for  future  test  flights.  Figure  3.1  shows  an  outline  of  this 
research’s  methodology,  which  will  be  discussed  in  detail. 


AnalyzeCurrentNCost  Benefit/  ^^Cost Sensitivity  >^ata Variability  \^Heuristic Risk  Decision 
Process  ^^Interviews  ^  Analysis  ^  InVSM  /  Analysis  ^  Rule 

Figure  3.1:  Research  Methodology  Overview 

The  use  of  these  techniques,  both  quantitative  and  qualitative,  offers  a  decision- 
driven  instead  of  hypothesis-driven  methodology.  A  decision-driven  methodology  is 
intended  to  facilitate  making  a  decision  about  the  applicability  or  worth  of  the  process  in 
a  given  situation  (Krathwohl,  1998).  Such  applicability  to  this  situation  may  include  the 
total  cost  of  the  SUAV,  its  total  impact  energy,  and  weight  of  the  vehicle.  This  overall 
heuristic  evaluation  should  offer  the  desired  consensus  or  accepted  basis  for  decision 
making.  For  example,  if  the  cost  of  the  SUAV  is  less  than  that  of  the  total  cost  the  test, 
then  it  may  be  necessary  to  reevaluate  the  total  cost  of  the  joint  AFRL/AFFTC  approval 
process.  Adversely,  if  the  cost  of  the  UAV  is  greater  than  that  of  the  total  cost  of  the 
flight  test  process  and  execution,  then  the  approval  process  is  likely  justified  for  greater 
accountability  and  safety  reviews.  This  approach  may  prove  to  be  useful  when  comparing 
the  current  UAV  flight  approval  process  to  an  abbreviated  process. 

3.2  Qualitative  Cost-Benefit  Analysis 

A  qualitative  cost-benefit  analysis  is  defined  as  the  determination  of  the  costs  of 
achieving  certain  benefits  (Krathwohl,  1998).  Cost-benefit  analysis  methodologies  have 
been  used  for  multiple  purposes  in  many  different  studies;  its  outcomes  must  be 
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translated  into  monetary  or  other  measure  of  value  so  their  relation  to  costs  may  be 
determined  (Krathwohl,  1998).  For  a  cost-benefit  analysis  conducted  in  this  research,  the 
benefit  may  be  in  the  form  of  cost  and  time  savings  from  an  abbreviated  flight  test 
process. 

According  to  Krathwohl  (1998),  cost-benefit  analyses  are  often  greatly  affected 
by  the  assumptions  of  difficult-to-estimate  indirect  costs  and  outcomes,  such  as  the 
personnel  costs  of  individuals  with  different  levels  of  qualifications.  By  measuring 
outcomes  across  different  programs  according  to  the  magnitude  of  their  effects  relative  to 
their  costs  in  comparable  terms,  cost-effectiveness  analysis  allows  a  comparison  of 
choices  of  achieving  certain  benefits  in  relation  to  other  methods  of  obtaining  the  same 
benefits  (Krathwohl,  1998;  Rossi  &  Freeman,  1985). 

3.2.1  Criteria  Development  and  Assumptions  for  Cost-Benefit  Analysis 

In  order  to  quantify  the  current  process  used  by  the  USAF  per  the  MO  A  between 
AFRL  and  AFFTC,  a  VSM  of  the  flight  test  approval  process  is  reused  from  a  lean 
process  improvement  study  conducted  by  Homan  and  colleagues  (2011),  as  shown  in 
Appendix  B.  To  ensure  this  research  quantifies  the  standard  USAF  flight  test  process  and 
any  abbreviated  flight  test  process  for  SUAVs,  a  comprehensive  holistic  assumption  was 
required  for  the  comparison  of  both  processes.  It  is  assumed  that  all  work  has  a  value- 
added  and  a  non-valued-added  component  to  it.  First,  any  value-added  (VA)  work 
identified  in  resources  and  manpower  is  assumed  to  be  valid — that  an  average  hourly  rate 
and  time  per  person  reviewing  the  flight  test  program  was  standard  across  all 
organizations.  Second,  any  non-value-added  (NVA)  work  time  and  resources  consists  of 
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waste  and  necessary  waste,  which  include  policy,  regulations,  and  standard  processing 
time.  Non-value-added  “pure  waste”  consists  of  the  “cost  of  delays,”  in  which  man-hours 
were  considered  “lost”  due  to  queuing  effects  in  the  approval  process  and  doing  “no 
work”  while  waiting. 

Furthermore,  other  assumptions  in  the  data  gathering  of  this  study  include  the 
value-added  work  in  terms  of  resources  and  manpower  for  SUAV  programs  and  those  of 
larger  manned  aircraft  are  likely  to  be  different  from  each  other.  It  was  assumed  that 
SUAV  programs  do  not  require  as  much  resources  and  manpower  as  those  of  larger 
manned  aircraft  programs.  Another  assumption  is  that  system  complexity  and  safety 
guidelines  of  both  types  of  vehicles  are  assumed  to  be  different  due  to  the  nature  of 
manned  versus  unmanned  aircraft  (i.e.,  manned  vehicles  are  more  complex  and  require  a 
higher  standard  of  safety  guidelines  than  SUAVs).  Lastly,  the  risk  assessment  and 
management  of  both  classes  of  vehicles  are  also  assumed  to  be  different — that  SUAVs 
are  sometimes  expendable  because  of  their  low  manufacturing  and  operating  costs  when 
compared  to  larger  conventional  aircraft. 

3.2.2  Notional  Balance  Statement 

By  using  the  criteria  for  qualitative  cost-benefit  analysis  and  the  comprehensive 
assumptions,  a  notional  overall  breakdown  of  the  total  cost  of  flight  testing  a  SUAV  is 
proposed  in  Table  3.1.  The  total  cost  of  a  SUAV  flight  test  includes  the  costs  of 
approving  flight  status  from  both  organizations  (AFRL  and  AFFTC)  and  the  cost  of  test 
execution. 
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Table  3.1:  Notional  SUAV  Flight  Test  Balance  Statement 


SUAV  Program  Asset  Cost 

Test  Execution  Cost 
AFRL  Review  Cost 

AFRL  Risk  Abatement  Cost 
Value  Added  Work 
Non-value  Added  Work 
Waste 

Necessary  Waste 
AFFTC  Review  Cost 

AFFTC  Risk  Abatement  Cost 
Value  Added  Work 
Non-value  Added  Work 
Waste 

Necessary  Waste 
Total  SUAV  Asset  and  Liabilities 


This  flight  test  approval  process  can  be  summarized  in  its  most  basic  and  simple 
balance  statement  equation  based  on  the  assumptions  and  criteria  discussed  earlier: 

Tsstjotal  Cost  Review  Cost  Cost  TE^osh  whcrC, 

AFRL  Review  Cost  =  AFRL  RAC  +  VA  +  NVA 
AFFTCReviewCost  =  AFFTCrac  +VA+  NVA 

The  Test  Total  Cost  is  the  total  cost  of  the  test.  The  Review  Costs  at  AFRL  and  AFFTC 
include  the  Risk  Abatement  Cost  (RAC)  and  the  value-added  (VA)  and  non-value-added 
(NVA)  costs.  The  Risk  Abatement  Cost  (RAC)  represents  the  safety  and  technical  reviews 
performed  by  the  approving  authority  at  AFRL  and  AFFTC  prior  to  test  execution.  The 
respective  VA  and  NVA  costs  at  AFRL  and  AFFTC  are  directly  and  indirectly  incurred 
from  both  organizations  as  part  of  approving  process  per  regulations  and  policy,  based  on 
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their  respective  resources  and  manpower.  Next,  TEcost  is  the  cost  of  test  execution,  which 
is  the  operating  cost  to  fly  the  SUAV.  Finally,  the  SUAV  Asset  Cost  consists  of  the 
vehicle  platform  cost  and  the  payload  cost.  However,  for  cost  simplicity,  this  research 
will  not  differentiate  between  these  two  costs  but  assumes  they  are  a  single  asset  cost. 

The  basic  notional  balance  statement  sheet  presented  may  be  elementary,  as  it 
only  shows  a  very  generic  cost  to  the  total  SUAV  flight  test  operation.  Additionally, 
when  comparing  the  two  balance  statements,  one  with  AFFTC  risk  abatement  cost  and 
the  other  without  AFFTC  risk  abatement  cost,  the  PM  would  choose  the  lesser  of  the  two 
for  cost  and  schedule  savings.  However,  further  analysis  in  risk  assessment,  data 
variability,  and  cost  sensibility  may  call  into  question  this  simple  approach. 

3.2.3  Interviews 

The  qualitative  cost-benefit  analysis  for  this  study  will  be  coupled  with  insights 
from  qualitative  interviews  in  order  to  elicit  the  processing  costs  of  an  abbreviated 
process.  It  is  assumed  that  the  cost-benefit  analysis  requires  personal  interviews  for  data 
validation  and  strengthening  the  results.  Interviews  are  one  qualitative  research 
methodology  that  allows  the  interviewer  to  gain  a  depth  of  responses  from  individuals  in 
the  process,  yet  it  has  its  own  disadvantages.  Interview  responses  are  often  reflected  by 
individual  biases,  and  may  also  be  in  a  form  of  confidentiality  unbeknownst  to  the 
interviewer  (Krathwohl,  1998).  These  responses  may  be  difficult  to  analyze  due  to  the 
sensitivity  of  the  subject  and  the  predisposition  of  individuals  to  respond  a  certain  way. 
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3.2.4  Cost  Sensitivity 

Cost  sensitivity  measures  how  variations  in  the  overall  [SUAV]  configuration 
affect  the  resource  requirements  and  costs  (Tenzer,  1965).  As  discussed  earlier,  there  are 
always  some  forms  of  uncertainties  and  risks  involve  in  flight  testing,  especially  to 
SUAVs  when  its  total  cost  is  sensitive  to  its  payload.  One  way  to  measure  cost  sensitivity 
is  the  design  and  operational  parameters  of  the  SUAV.  The  total  flight  test  costs  may 
greatly  increase  if  an  advanced  sensor  payload  is  attached  to  the  SUAV,  which  may  also 
significantly  increase  the  risk  factor  and  the  costs  associated  with  the  whole  program. 
Another  way  to  look  at  cost  sensitivity  is  the  question  of  necessity  in  redundant  reviews 
for  SUAVs.  SUAV  flight  parameters  and  their  associated  costs  may  be  the  driving  factor 
in  adding  the  flight  approval  review  times,  as  these  reviews  may  cause  delays  to  the 
programs.  To  an  extent,  any  changes  in  the  associated  costs  of  the  program  are  sensitive 
to  the  decision  rule  used. 

3.3  Data  Variability  Analysis 

In  addition  to  the  qualitative  cost-benefit  analysis,  personal  interviews,  and  cost 
sensitivity  analysis,  data  variability  obtained  using  the  VSM  will  be  analyzed.  A  VSM  is 
one  the  first  products  created  when  attempting  a  process  improvement  as  described  by 
Womack  and  Jones  (2003).  McManus  (2005)  states  that  a  VSM  is  a  tool  for  achieving  an 
efficient  process,  and  that  its  sole  purpose  is  for  process  improvement  by  evaluating  the 
process  holistically.  The  goal  of  the  VSM  is  to  depict  material  and  information  flows 
across  and  throughout  all  the  value-adding  and  non- value  adding  (waste)  processes 
required  to  produce  the  product  (i.e.,  approval  status)  and  deliver  it  to  the  customer  (i.e.. 
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the  program  manager  and  test  team)  (McManus  &  Millard,  2003).  Therefore,  the  VSM  is 
an  appropriate  approach  due  to  its  “all-in-one”  top-level  picture  of  all  the  processes  that 
occur  in  the  flight  approval  process,  from  the  time  a  test  plan  is  submitted  to  the  time  it  is 
approved  for  flight  status. 

In  a  perfect  process,  every  link  in  the  approval  chain  should  be  synchronized, 
making  the  approval  system  as  lean  as  possible.  However,  achieving  a  high  level  of 
synchronization  requires  a  “smooth  flow”  process  (Murman,  2002).  The  current  flight 
test  approval  process  involves  moderate  to  high  variability  in  the  value  flow  streams,  as 
organizations  have  variable  flight  test  demands  at  different  times  and  resources  to 
approve  fight  tests.  According  to  Oehmen  and  Rebentisch  (2010),  variability  in  process 
execution  leads  to  widely  varying  process  times  and  thus  a  low  degree  of 
synchronization.  Oehmen  and  Rebentisch  (2010)  also  argue  that  over  utilized  capacities 
(e.g.,  engineers)  can  also  be  a  cause  for  such  variations,  as  these  disturbances  cannot  be 
buffered  and  immediately  lead  to  delays. 

One  of  the  purposes  of  VSM  is  to  identify  and  predict  locations  of  process 
variability.  If  variability  cannot  be  controlled  due  to  budget  or  schedule  constraints  in  the 
program,  then  there  should  be  enough  takt  time  in  the  individual  process  steps  so  the  end 
step  in  the  process  will  not  have  high  variability.  Takt  time  is  the  number  of  units  of  work 
demanded  by  the  customer  divided  by  the  available  time  (McManus,  2005);  for  this 
research  effort,  it  is  the  number  of  days  needed  for  a  completed  test  plan  review  to  get 
approved  for  flight  test.  According  to  McManus  (2005),  the  single  most  important 
concept  of  lean  product  development  is  achieving  a  takt  time,  and  that  all  tasks  or  steps  in 
the  process  must  then  proceed  at  this  pace. 
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For  example,  if  the  FOA’s  average  rate  for  approving  a  test  plan  is  three  days, 
then  the  takt  time  at  the  FOA  step  is  three  even  if  the  aetual  data  varies  with  the 
workload.  The  FOA  must  be  able  to  support  this  average  takt  time  of  three  days  per  test 
plan  review,  or  the  overall  demand  for  work  cannot  be  met,  which  would  trigger 
bottlenecks  and  delays  in  the  downstream  processes.  In  summary,  in  an  uncontrolled 
process  with  variations  of  any  sort,  queuing  effects  will  tend  to  build  delays  between 
process  steps  and  pile  up  work  at  bottlenecks  (McManus,  2005). 

3.4  Managing  Acceptable  Cost  and  Optimal  Risk 

Many  in  the  T&E  community  know  the  risk  hazards  present  in  the 
operation  of  any  complex  system,  manned  and  unmanned  systems  included.  This  concern 
is  probably  one  of  the  major  reasons  why  UAV  standards  and  regulations  are  being 
worked  throughout  many  organizations  in  the  development  of  UAV  safety  and 
airworthiness  certification.  Based  on  data  from  the  T&E  community,  especially  those 
coming  from  the  military,  UAVs  have  a  poor  safety  record  (De  Garmo,  2004).  According 
to  an  April  2003  Congressional  Research  Service  report,  the  UAV  accident  rate  is  100 
times  that  of  manned  aircraft  (Bone  &  Bolkcom,  2003).  Another  study  conducted  by  the 
USAF  in  2002  concluded  that  the  accident  rate  for  UAVs  is  50  times  greater  than  that  of 
an  F-16  fighter  jet  (De  Garmo,  2004). 

A  more  recent  2009  DoD  Unmanned  Systems  Integrated  Roadmap  determined 
that  UAVs  have  suffered  mishaps  at  one  to  two  orders  of  magnitude  greater  than  the  rate 
(per  100,000  hours)  incurred  by  manned  military  aircraft.  Figure  3.2  shows  the  mishap 
rates  per  100,000  flight  hours  between  manned  and  unmanned  vehicles  between  1986  to 
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2006.  Notice  that  SUAVs  such  as  the  Hunter,  Shadow,  and  Pioneer,  whose  wingspans 
and  body  lengths  have  an  average  from  10  feet  to  20  feet,  have  higher  mishap  rates 
compared  to  those  of  other  types  of  vehicles.  Also  noted  is  the  decrease  in  mishaps  as 
these  SUAVs  accumulate  more  flight  hours.  The  decrease  in  SUAV  mishaps  shown  in 
Figure  3.2  represents  a  steep  learning  curve  from  SUAVs  as  the  technology  matures. 


Figure  3.2:  U.S.  Military  Aircraft  and  UAV  Class  A  Mishap  Rates 
(DoD  FY2009-2034  Unmanned  Systems  Integrated  Roadmap,  2009) 


It  is  for  this  very  reason  of  UAV  mishap  rates  that  PMs  and  UAV  operators 
should  carefully  manage  the  cost  of  these  SUAVs  and  reduce  the  risk  to  an  optimal  level 
necessary  for  the  continuation  of  flight  tests  in  the  program.  As  clearly  stated  in  the 
previous  chapter  (Chapter  2:  Literature  Review),  the  primary  goal  of  any  flight  approval 
process  for  airworthiness  and  flight  certification  is  to  reduce  risks  and  increase  safety 
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standards  among  SUAVs  and  other  airborne  vehicles,  including  those  who  flight  test 
them.  While  it  has  always  been  the  PM’s  goal  to  reduce  the  costs  in  UAV  flight 
operations  and  any  overhead  costs  incurred  while  waiting  for  the  flight  approval  process, 
the  cost  of  risk  abatement  in  time  and  resources  is  almost  always  never  explicitly 
managed.  However,  with  limited  resources  and  time,  the  question  becomes,  “How  much 
risk  abatement  is  enough  as  part  of  the  flight  test  approval  process?  ”  Figure  3.3  shows  a 
classical  answer  from  economics  where  the  optimal  level  of  risk  is  where  the  sum  of  the 
cost  of  risk  abatement  and  the  expected  losses  from  risk  is  at  a  minimum  (Morgan,  1981). 


Figure  3.3:  Managing  Cost  and  Level  of  Risk 
(Morgan,  1981) 


Two  of  the  most  common  forms  of  risks  are  “acceptable  risk”  and  “optimal  risk.” 
(Morgan,  1981).  Acceptable  risk  implies  a  predetermined  criterion  or  standard  for  a 
maximum  risk  threshold  level  below  which  risk  will  be  tolerated,  and  thereby  permits  the 
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evaluation  of  cost,  national  priority  interests,  and  number  of  tests  to  be  conducted  (U.S. 
Army  White  Sands  Missile  Range,  Morgan).  Optimal  risk  implies  a  trade-off  that 
minimizes  the  sum  of  all  undesirable  consequences  (Morgan,  1981).  It  is  this  optimality 
that  justifies  careful  attention,  thus  the  basis  of  this  research  in  terms  of  an  abbreviated 
approval  process  for  SUAVs  where  redundancy  in  the  cost  of  risk  abatement  may  be 
unnecessary. 

One  way  to  reduce  the  cost  of  risk  abatement  is  to  actually  reduce  its  cost  by 
shortening  some  of  the  safety  review  times.  However,  reducing  the  cost  of  risk  abatement 
in  this  way  would  likely  increase  the  level  of  risks  and  the  expected  losses  from  risk, 
resulting  in  an  increase  to  the  total  cost  of  the  program  because  of  the  accompanying 
reduction  in  scrutiny  of  the  safety  review.  Therefore,  the  primary  method  used  in  this 
study  is  to  determine  the  cost  benefit  of  an  abbreviated  flight  test  to  achieve  an  optimal 
level  of  risk.  This  optimal  level  of  risk  should  demonstrate  an  appropriate  reduction  in  the 
cost  of  risk  abatement  while  maintaining  or  decreasing  the  expected  losses  from  risk,  thus 
minimizing  the  total  costs  incurred  from  the  flight  approval  process  and  UAV  operation. 
In  investigating  the  notional  optimal  level  of  risk,  this  study  proposes  shifting  the  total 
cost  curve  and  the  cost  of  risk  abatement  curve  to  the  left,  as  seen  in  Figure  3.4. 

If  a  reduction  in  the  cost  of  risk  abatement  is  attainable,  either  through  eliminating 
AFFTC  reviews  or  decreasing  the  complexity  of  the  vehicle,  the  abbreviated  approval 
process  may  reveal  that  it  can  be  condensed  and  shortened  with  probable  cost  savings  to 
the  total  cost  and  expected  losses  from  risk.  Various  features  in  the  flight  test  approval 
process  can  be  factored  in  when  deciding  how  much  risk  abatement,  or  for  the  purposes 
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of  this  research,  the  degree  to  which  safety  reviews  are  modulated  in  the  process  to  save 
time  and  money. 


Cost 


Level  of  Risks 


Figure  3.4:  Notional  Optimal  Level  of  Risk 


3.5  Summary 

This  chapter  has  detailed  the  methodology  used  in  this  study.  The  notional  flight 
test  balance  sheet  with  proposed  assumptions  tailored  by  interviews  and  cost  sensitivity 
analysis  establish  the  conditions  needed  to  create  a  better  understanding  of  the  costs 
associated  with  the  flight  test  approval  process.  Furthermore,  data  variability  analysis  will 
be  used  to  present  a  thorough  overview  of  the  current  process  depicted  in  the  VSM  and 
the  issues  that  arise  from  it.  Additionally,  by  combining  the  balance  sheet  and  the 
analyses,  this  methodology  seeks  to  expose  the  advantages  and  disadvantages  of  flight 
test  process  steps  in  the  overall  chain  of  reviews  and  approvals.  Finally,  this  chapter 
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notionally  identified  the  costs  of  risk  abatement  and  the  expected  losses  due  to  SUAV 
flight  tests.  The  results  and  analysis  using  this  methodology  are  presented  in  Chapter  4. 
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IV.  Results  and  Analysis 


This  chapter  describes  the  results  and  analysis  of  the  data  gathered  from  the  flight 
test  offices.  The  first  section  details  the  flight  test  balance  statement  with  relevant 
assumptions  in  operational  and  personnel  costs  to  determine  the  cost-benefit  of  the 
review  process  at  AFFTC.  The  second  section  focuses  on  VSM  analysis  of  cost 
sensitivity  and  time  variability  in  the  chain  of  approval  process  through  a  Variability 
Simulator.  Next,  the  analysis  on  risk  management  with  acceptable  risk  to  costs  and 
optimal  risk  in  flight  testing  are  shown.  These  cost  benefits  are  then  measured  against  the 
cost  of  risk  abatement  and  expected  losses  to  the  UAV  to  determine  the  threshold  where 
it  makes  sense  that  the  review  process  at  AFFTC  adds  value. 

4.1  Cost  Measurement  Analysis 

Cost  measurements  are  taken  from  labor  hours  and  costs  of  operating  and 
maintaining  a  SUAV.  The  intention  of  this  analysis  is  to  determine  whether  the  extra 
reviewing  and  approval  process  at  AFFTC  is  needed  from  the  point  of  view  of  a 
simplistic  SUAV  flight  test.  These  costs  were  taken  from  historical  data  and  labor  hours 
were  assumed  based  on  the  rank  and  pay  grade  of  the  officer  in  charge  of  certain 
processes.  Comparisons  were  made  with  significant  assumptions  due  to  the  sensitivity  of 
the  SUAV  costs  and  the  difficult-to-estimate  indirect  costs.  Nevertheless,  the  balance 
statement  in  the  cost  measurement  analysis  shows  the  financial  resources  available  so  that 
PMs  may  decide  how  to  use  cost  information  in  the  assessment  of  economic  alternatives. 
It  may  sound  elementary  that  the  lower  cost  in  the  balance  statement  will  most  likely  be 
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chosen,  but  it  may  be  apparent  that  there  are  several  other  issues  and  problems  that  must 
be  dealt  with  in  order  to  develop  further  analysis  for  cost  benefits. 

4.1.1  Estimated  Cost  Balance  Statement 

Historieal  data  from  201 1  shows  22  different  SUAV  tests  within  the  continental 
United  States,  averaging  $66,500  in  eosts  per  vehiele.  Each  program  had  an  average  of 
two  SUAVs  produced  and  flown  during  the  life  of  the  program,  whieh  ean  be  added  up  to 
a  total  of  $133,000  in  SUAV  program  assets.  Eaeh  SUAV  also  had  an  average  of  23.8 
sorties  and  9.9  flight  hours  (see  Appendix  C  for  SUAV  costs  and  mishap  data).  The  test 
execution  cost,  estimated  at  $30,000,  was  based  off  of  the  average  range  cost,  fuel  eost, 
and  any  extra  burden  and  overhead  eosts  including  personnel  involved  in  the  flight  test  of 
the  SUAV.  The  following  proeessing  days  were  taken  from  the  eurrent  VSM  flight  test 
proeess  (Appendix  B)  developed  by  Homan  et  al.  (201 1)  and  the  AFRL  T&E  Office 
(Appendix  C)  to  determine  a  eonservative  estimated  eost  of  the  review  proeess. 

The  AFRL  teehnical  and  safety  reviews  (TRB  &  SRB)  typically  took  four  to  five 
weeks  (25  working  days)  prior  to  the  aetual  test  exeeution;  therefore,  it  was  possible  to 
estimate  the  value-added  {VA)  time  and  costs  of  the  review  process.  With  a  conservative 
assumption  of  a  four-member  team  ineluding  those  at  the  AFRL  T&E  Offiee  working  25 
working  days  and  one  Lieutenant  Colonel  (0-5)  or  a  GS-15  pay  grade  teehnieal  lead 
working  two  (2)  days,  the  total  TRB  and  SRB  cost  averaged  $20,000.  Sinee  AFFTC  does 
not  normally  take  part  in  the  TRB  and  SBR  at  AFRL  for  low  risk  test  plans,  this  value 
was  low;  but  nevertheless,  an  estimate  of  one  (1)  working  day  was  added  for  any 
comments  provided  from  the  FOA  to  the  AFRL  TRB  and  SRB. 
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Next,  the  non-value  added  (NVA)  work  (waste  and  necessary  waste)  both  at 
AFRL  and  AFFTC  were  greatly  assumed  as  standard  processing  time  with  delays.  These 
delays  can  sometimes  be  considered  as  buffer  times  or  variations  against  uncertainties  in 
the  overall  process.  The  assumption  on  AFRL  waste  and  necessary  waste  was  one  (1)  and 
two  (2)  working  days,  respectively.  AFFTC ’s  non-value  added  waste  work  was  assumed 
to  be  two  (2)  working  days. 

Finally,  the  AFRL  risk  abatement  cost  (AFRLrac)  was  also  based  on  an  average 
0-5  or  a  GS-15  pay  grade  combined  with  the  current  five  (5)  working  days  to  approve  a 
flight  test  at  AFRL  per  the  VSM.  Likewise  with  AFFTC  risk  abatement  cost 
(AFFTCrac),  except  the  current  average  processing  time  to  approve  an  AFRL  flight  test 
at  AFFTC  was  six  (6)  working  days,  which  was  based  off  of  the  data  provided  by  the 
AFRL  T&E  Office.  When  combining  all  of  this  data  together,  the  total  average  SLAV 
asset  and  liability  has  a  total  cost  of  $191,500. 

Table  4.1  summarizes  the  value-added  (VA)  time  required  at  critical  steps  through 
the  flight  approval  process.  The  first  step  requires  the  PM  to  meet  with  the  AFRL  T&E 
Office  to  conduct  TRB  and  SRB  and  document  mishap  reporting  procedures  in  the  test 
plan — all  of  which  happen  four  to  five  weeks  prior  to  test  execution.  The  second  step 
requires  the  AFRL  Technical  Director  Division  Chiefs  (GS-15)  review  and  approval  to 
start  test,  which  normally  takes  five  (5)  working  days.  The  third  step,  which  is  in 
conjunction  with  the  second  step,  requires  the  FOA  at  AFFTC  to  review  and  approve  the 
test  plan  in  order  to  start  the  flight  test,  which  averages  six  (6)  working  days.  Table  4.2 
shows  the  balance  statement  of  an  average  typical  SUAV  flight  test. 
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Table  4.1:  Estimated  Time  for  AFRL/AFFTC  Approval  Review 


Process  Step 

AFRL  Processing 
Time  (Work  days) 

AFFTC  Processing 
Time  (Work  days) 

PM  meets  with  AFRF 
T&E  Office  to  conduct 
TRB  &  SRB  and 
finalizes  test  plan 

4  members  take  25  WD 

1  0-5/GS-15  take  2  WD 

1  0-5/GS-15  take  1  WD 

Get  AFRF  Division 
Chief  (GS-15)  to  review 
and  approve  to  start  test 

1  0-5/GS-15  take  5  WD 

n/a 

Get  AFFTC  FOA  (0-5) 
to  review  and  approve  to 
start  test 

n/a 

1  0-5/GS-15  take  6  WD 

Table  4.2:  Notional  SUAV  Cost  Balanee  Statement 


SUAV  Program  Asset  Cost  $133,000 

Test  Execution  Cost  $30,000 

AFRL  Review  Cost 

AFRL  Risk  Abatement  Cost  $2, 1 00 

Value  Added  Work  $20,000 

Non-value  Added  Work 

Waste  $1,200 

Necessary  Waste  $1 ,500 

AFFTC  Review  Cost 

AFRL  Risk  Abatement  Cost  $2,500 

Value  Added  Work  $400 

Non-value  Added  Work 
Waste  $800 

Necessary  Waste  $0 


Total  SUAV  Asset  and  Liabilities  $191,500 


TcStjotal  Cost  AFRL 


'Review  Cost 


+ AFFTQ 


Review  Cost 


+  TE. 


cost 
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4.1.2  Initial  Cost  Results 


As  previously  emphasized,  it  is  important  to  recognize  that  these  monetary  values 
are  presented  with  significant  assumptions  based  on  empirical  observations  of  only 
somewhat  similar  situations  befitting  an  exploratory  research  such  as  this.  From  the  data 
presented,  with  an  average  total  value  of  $191,500  in  SUAV  asset  and  liabilities,  the  total 
AFFTC  review  costs  ($3,700)  did  not  represent  any  substantial  impact  to  the  total  value 
of  the  SUAV.  Not  only  was  the  AFFTC  review  cost  less  than  anticipated,  the  average 
reviewing  time  was  much  shorter  than  previously  expected. 

With  regards  to  flight  mishap  data  from  the  AFRL  T&E  Office  (Appendix  C), 
SUAV  flights  from  2011  had  nine  flights  mishaps  with  a  total  of  over  $625,000  in 
damages,  which  translated  into  an  average  of  $69,400  per  flight  mishap.  Additionally, 
data  from  the  AFRL  T&E  Office  showed  an  average  of  five  (5)  working  days  for  a  return 
to  flight  approval  from  AFFTC  upon  learning  of  a  mishap.  In  two  mishap  instances,  the 
return  to  flight  processing  time  AFFTC  took  to  approve  was  three  working  days  and  six 
working  days,  respectively.  By  using  similar  assumptions  and  estimates  on  cost,  these 
five  working  days  had  an  estimated  process  cost  of  a  little  over  $2,000  for  a  return  to 
flight  approval  from  AFFTC.  Again,  the  average  time  and  cost  of  a  return  to  flight 
approval  from  AFFTC  was  insignificant  compared  to  the  total  cost  in  mishap  damages. 

In  summary,  the  cost  measurement  analysis  from  the  data  given  suggested  that  the 
process  time  and  process  cost  from  the  AFFTC  review  was  considerably  less  than 
previously  expected.  Moreover,  the  time  to  approve  a  return  to  flight  following  a  mishap 
was  reasonably  less  in  terms  of  cost  and  time.  This  initial  cost  analysis  shows  that  the 
AFFTC  involvement  in  AFRL  flight  tests  may  be  beneficial  and  instrumental  in 
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mitigating  AFRL  flight  mishaps,  considering  SUAV  flight  tests  in  201 1  had  experieneed 
a  mishap  with  over  $69,000  in  damages  for  eaeh  mishap.  The  extra  review  at  the  AFFTC 
may  cause  additional  time  and  overhead  costs  for  the  test  management  organization,  but 
in  the  long  run,  it  may  be  valuable  in  providing  eomments  and  insights  to  the  PMs  and 
the  test  teams  at  AFRL,  thereby  possibly  redueing  future  flight  mishaps.  This  perceived 
value  of  extra  small  dollars  and  time  to  “buy”  a  decreased  ehanee  of  failure  may  be 
enough  to  convinee  the  PMs  to  spend  the  additional  funds  to  mitigate  mishaps.  However, 
given  the  nature  of  R&D  in  UAV  test  flights,  mishaps  that  cost  a  few  hundred  to  a  few 
thousand  dollars  are  sometimes  eonsidered  negligible  and  acceptable  in  favor  of 
improved  researeh. 

Additional  interviews  were  eonducted  with  the  AFRL  T&E  Offiee  as  more  details 
on  AFFTC  review  process  were  revealed.  One  of  the  primary  reasons  why  the  approval 
proeessing  time  from  AFFTC  was  eut  shorter  than  what  was  anticipated  at  the  beginning 
of  this  research  was  the  personnel  ehanges  that  took  place  in  the  412*  OG  at  AFFTC 
since  the  Homan  et  al.  (201 1)  study.  Personnel  at  the  AFRL  T&E  Offiee  notieed  the  trend 
in  approval  review  was  slashed  to  almost  half  that  from  previous  submissions.  The  AFRL 
T&E  Offiee  also  notieed  there  were  no  additional  re-work  or  additional  value  added 
feedback  from  the  FOA.  This  decreasing  trend  in  approval  time  and  the  laek  of  in-depth 
reviews  from  AFFTC  suggested  either  an  absence  of  interest  in  SUAV  flight  test  reviews 
from  the  new  personnel,  or  that  the  new  FOA  has  come  to  fully  trust  the  flight  test 
organizations  at  AFRL  with  simple  SUAV  tests  and  designs.  In  addition  to  the  new  FOA 
personnel  at  AFFTC,  the  AFRL  T&E  Offiee  also  weleomed  in  a  new  Chief,  who  is  a  Test 
Pilot  Sehool  (TPS)  graduate  from  Edwards  AFB.  Both  organizations  have  seen  an 
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increase  in  direet  communieation  on  issues  relating  to  SUAV  tests.  Perhaps  the 
experienced  TPS  graduate  and  new  Chief  at  the  AFRL  T&E  Office  brings  new  skills  or 
“value-added”  to  the  process,  henee  the  deereased  time  in  the  approval  process. 

4. 1. 3  Cost  Sensitivity  Results 

There  is  difficulty  in  estimating  eost  evaluations,  especially  given  sensitivities 
associated  with  the  programs.  These  sensitivities  include  the  design  and  operational 
parameters  of  the  SUAVs.  As  shown  in  Figure  4.1,  of  the  22  SUAV  flight  test  programs 
in  201 1,  nearly  half  were  new  and  prototypes  with  variable  costs  to  each  program,  none 
of  which  are  dependent  on  their  system  maturity. 


Mature  New  Prototype 

System  Maturity 


Figure  4.1:  System  Maturity  vs.  Cost 

Figure  4. 1  also  shows  that  costs  are  not  sensitive  to  system  maturity,  but  rather  to 
the  payloads  earned  by  these  SUAVs.  The  cost  of  some  of  these  SUAVs  ean  be  high, 
mostly  due  to  their  expensive  payloads  such  as  radar  sensors  and  imaging  capabilities. 
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Since  the  initial  results  and  mishap  data  have  shown  high  vulnerability  for  SUAV 
mishaps  at  an  average  of  $69,400  in  losses,  PMs  and  test  teams  are  reeommended  to  have 
initial  estimates  on  the  program’s  feasibility  and  eategorize  test  requirements  in  phases  of 
testing  in  order  to  reduee  risks  to  program  cost.  These  phases  of  testing  and  development 
should  be  separated  into  a  list  of  “must  haves”  and  “good  to  haves”  with  user 
prioritization  that  can  be  used  to  conduct  trade-off  at  later  stages  of  aequisition 
development.  These  phases  of  testing  and  development  should  minimize  eost 
uncertainties,  whereas  an  “all-in-one”  package  (i.e.,  one  expensive  payload  on  a  SUAV) 
may  present  a  higher  risk  to  cost  should  a  mishap  occur.  This  recommendation  is  in 
agreement  with  the  industrial  practice  of  involving  customers  early  in  program 
requirements  with  elear  communication  to  separate  their  needs  into  “must  haves”  and  a 
“wish  list”  (CNSTAT  et  al.,  2011).  CNSTAT  et  al.  (201 1)  also  reeognizes  some 
flexibility  that  allows  for  modifications  responsive  to  users’  needs  and  changing 
environments.  However,  any  ehanges  to  program  requirements  can  result  in  considerable 
cost  increases  and  sehedule  delays. 

A  substantial  part  of  the  many  levels  of  safety  reviews  in  the  USAF  is  the 
structure  of  aeeountability  in  the  design  and  development  of  the  system.  A  system  that 
has  mature  technologies  will  likely  experienee  safer  operation  than  those  that  have 
immature  systems.  With  SUAV  payload  costs  that  sometimes  exceed  the  cost  of  the 
vehicle,  it  becomes  a  question  of  technology  maturity  of  the  SUAV  itself  Teehnology 
immaturity  such  as  those  of  SUAVs  eoupled  with  an  expensive  payload  will  certainly 
cause  scrutiny  in  reviews.  According  to  CNSTAT  et  al.  (2011),  teehnology  immaturity  is 
known  to  be  a  primary  eause  of  schedule  slippage  and  cost  growth  in  DoD  program 


55 


acquisition.  The  success  of  a  system  maturity  strategy  depends  on  the  consistent  and 
continuous  definition  of  requirements  and  the  maturation  of  technologies.  This 
“evolutionary  acquisition”  or  a  spiral  acquisition  approach  to  transition  new  technology 
to  operational  capability  is  reflected  in  the  DoD  Instruction  5000.2.  The  objective  of 
evolutionary  acquisition  is  to  balance  the  needs  and  available  capability  with  resources, 
and  to  put  capability  into  the  hands  of  the  user  quickly  (DoD  AT&L). 

In  summary,  system  maturity  costs  were  varied  with  SUAV  performance  and 
payload  cost.  To  minimize  large  mishap  costs,  a  priority  list  of  “must  haves”  and  “good 
to  haves”  should  be  implemented  and  decided  on  what  and  when  to  test  first  instead  of 
putting  all  eggs  in  one  basket  and  risk  it  all  from  system  failure. 

4.2  Value  Stream  Mapping  Analysis 

Value  Stream  Mapping  (VSM)  examines  the  business  process  and  seeks  to 
understand  and  improve  such  efforts.  While  this  study  does  not  focus  on  the  efficiency  of 
the  process,  it  does  examine  the  necessity  of  the  extra  reviews  at  AFFTC  and  the  cost 
benefit  of  having  such  reviews.  As  Homan  et  al.  (2011)  described  in  the  present-state 
VSM,  it  was  apparent  that  one  of  the  larger  areas  of  waste  was  the  specific  loops  between 
the  different  stakeholders  (i.e.,  program  managers)  and  the  AFRL  T&E  Office.  Other 
problems  revealed  by  Homan  et  al.  (201 1)  included  the  lack  of  initial  notifications  of 
pending  tests  and  the  absence  of  standardization  in  the  test  plans  when  submitted  for 
approval.  The  five  main  problems  identified  by  Homan  (2011)  were: 

1.  Late  notification  and/or  non-standard  work  (test  plans  or  safety  packages) 
by  PMs 

2.  Firefighting  and  heroics  by  AFRL  T&E  Office 
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5.  Bottleneck  at  FOA  (high  throughput  and  low  capacity) 

4.  Lack  of  expected  target  timeline  and  non-standard  work  (mishap  report) 
for  return-to-flight  process 

5.  Lack  of  transparency  throughout  processes  for  all  stakeholders 

In  quantifying  the  present-state  VSM  and  the  cost  measurements  which  were 
discussed  in  the  previous  result,  initial  data  suggested  that  AFFTC  involvement  in  AFRL 
flight  tests  may  be  beneficial  in  providing  feedback  and  mitigating  mishaps  given  that  the 
costs  and  time  required  were  miniscule  compared  to  the  total  cost  of  the  UAV  program. 
However,  further  analysis  was  needed  to  verify  and  determine  the  threshold  at  which 
AFFTC  involvement  in  low  risk  test  plans  adds  value  and  that  the  extra  review  is 
desirable  even  though  it  presents  a  fairly  minimal  investment  in  time  and  cost  to  the 
program.  Conversely,  this  threshold  will  also  need  to  show  that  AFFTC’s  involvement  in 
low  cost  programs  are  sometimes  inapplicable  because  of  SUAV  costs  and  complexity, 
and  any  program  that  falls  below  this  threshold  may  not  require  additional  reviews  at 
AFFTC. 

4.2.1  Data  Variability 

In  this  portion  of  the  analysis,  a  Variation  Simulator  developed  by  the  Lean 
Advancement  Institute  (LAI)  at  the  Massachusetts  Institute  of  Technology  (MIT)  Lean 
Academy  was  used  to  measure  the  impacts  of  variation  on  cycle  time  in  the  approval 
process.  The  intention  of  the  variability  simulation  is  to  assess  the  impact  that  variability 
has  on  process  performance.  Any  variability  in  the  simulation  value  is  used  for  the 
purpose  of  predicting  relative  differences  in  task  execution  at  a  particular  step.  Such 
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simulation  is  not  designed  to  east  judgment  or  predict  performance  at  AFFTC  or  AFRL 
T&E  Office. 


Reducing  process  variation  is  a  key  component  in  process  improvement 
methodologies  such  as  lean  Six  Sigma.  In  an  ideal  process  flow,  where  the  input  rate  is 
equal  to  or  less  than  that  of  the  output  rate  at  every  task,  the  flow  should  not  experience 
any  delays  and  the  process  is  deemed  fully  utilized.  However,  when  variations  in  the 
process  are  considered,  the  process  begins  to  experience  delays  and  work  begins  to 
accumulate  at  various  tasks.  For  the  sake  of  simplicity,  the  process  flow  begins  when  the 
PM  submits  a  draft  test  plan  to  the  AFRL  T&E  Office,  which  then  begins  the  safety  and 
technical  coordination  approval  process  internally  at  AFRL  and  externally  at  AFFTC. 
Once  the  approval  process  is  complete,  test  execution  can  begin. 

As  demonstrated  in  Figure  4.2,  the  simulation  assumed  no  output  rate  variations  at 
the  AFRL  T&E  Office  nor  the  review  authority  panels;  however,  the  PM  station  had 
variations  per  day  in  submitting  test  plans  to  the  AFRL  T&E  Office.  These  test  plans 
were  varied  from  one  to  five  per  day,  which  could  represent  a  lack  of  notifications  of 
pending  tests  and  lack  of  standardization  in  the  test  plans.  These  variations  coming  from 
the  PM  had  the  most  direct  impact  on  the  AFRL  T&E  Office.  Hence,  the  workload 
depicted  in  Figure  4.2  shows  the  accumulated  test  plans  at  the  AFRL  T&E  Office  over 
time  when  its  capacity  could  only  produce  three  test  plans  per  day. 


58 


PM 

.A. 

T&E  Office 

.A. 

AFRL  Reviews 

-A. 

AFFTC  Reviews 

-A. 

Execution 

(D)  0,1 ,2, 3, 4, 5 

A 

(C)  Always  3 

A 

(C)  Always  3 

A 

(C)  Always  3 

A 

(C)  Always  3 

Figure  4.2:  Variation  at  Program  Management 


Homan  et  al.  (2011)  mentioned  that  the  AFRL  T&E  Office  was  unable  to 
eliminate  the  constant  late  notifications  or  tardy  test  packages  arriving  from  the  program 
manager,  in  addition  to  the  non-standardized  test  packages.  Homan  et  al.  (2011)  also 
discussed  the  effect  of  “firefighting”  at  the  AFRL  T&E  Office  where  members  defer 
resources  to  problematic  (or  late)  programs  and  work  harder  to  meet  requirements  and 
push  completed  programs  to  the  next  station.  This  effect  causes  the  information  to  get 
“pushed”  instead  of  “pulled.”  According  to  McManus  (2005),  information  flow  should  be 
pulled,  not  pushed.  In  other  words,  information  should  flow  when  it  is  needed  by 
downstream  processes,  not  when  upstream  processes  are  done.  Repenning  et  al.  (2001) 
also  added  that  occasional  “firefighting”  may  offer  short-term  gains,  but  these  temporary 
gains  will  come  at  the  expense  of  the  long-run  flow  of  the  entire  process.  Firefighting  can 


59 


also  create  bottlenecks  for  the  downstream  process,  especially  when  the  later  process 
steps  have  variation,  which  possibility  is  discussed  in  the  next  simulation. 

As  shown  in  Figure  4.3,  the  FOA  at  AFFTC  had  variations  in  approving  SUAV 
test  plans  coming  from  AFRL.  One  of  the  main  problems  identified  earlier  was  the 
bottleneck  at  the  FOA  in  the  overall  process  (Homan  et  ah,  201 1).  This  bottleneck  was 
caused  by  the  increasing  number  of  test  packages  that  were  pushed  to  the  FOA  from  the 
AFRL  T&E  Office,  resulting  in  variations  in  approval  time.  In  many  instances  where 
programs  were  deemed  low  risk,  only  a  single  approval  authority  at  the  AFFTC  handled 
all  of  the  test  packages  from  AFRL.  The  problem  was  even  worse  when  the  single  FOA 
at  AFFTC  was  also  responsible  for  nine  other  local  test  organizations.  These  notional 
results  show  that  the  pushed  information  was  not  needed  by  the  recipient  in  the 
downstream  process,  and  it  would  certainly  not  flow  “at  the  right  time”  for  PMs  to 
execute  their  tests  on  schedule. 
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Figure  4.3:  Bottleneck  at  AFFTC 


In  all  simulations  conducted  with  variations  of  at  least  one  task,  the  test  execution 
schedule  was  greatly  affected.  The  simulation  could  continue  with  multiple  variables  at 
different  tasks,  but  the  purpose  was  to  demonstrate  the  process  behavior  over  a  long 
period  of  time.  With  reduced  input  variability  and  process  variability,  the  simulation 
shows  improvement  in  the  flow  process  and  allows  stations  to  have  excess  capacity  to 
pull  work.  Eliminating  variability  also  allows  straight-through  flow  to  meet  customer 
(i.e.,  program  managers)  demands. 

McManus  (2005)  specifies  that  bottlenecks  can  be  caused  by  effects  external  to 
the  process,  such  as  the  lack  of  resources,  information,  or  authority  to  proceed.  Such 
effects  coincide  with  the  current-state  of  the  AFRL/ AFFTC  approval  process.  That  is,  if 
there  is  a  lack  of  resources,  typically  the  FOA  personnel  are  not  available  to  start  a  task 
and  the  task  waits,  thus  creating  a  bottleneck.  If  the  externally-provided  information  is 
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not  available,  typically  because  of  the  inadequate  test  plans  from  the  PM,  the  tasks  either 
wait  or  proceed  with  “bad”  information  (i.e.,  firefighting)  only  to  be  reworked  later  (after 
mishaps).  Finally,  if  external  approval  authority  other  than  from  AFFTC  is  required  to 
proceed,  and  it  is  not  forthcoming,  the  task  will  wait. 

4.2.2  Value  Stream  Mapping  Results 

In  summary,  the  results  from  VSM  analysis  conducted  with  data  variability 
support  the  five  major  problems  which  were  first  identified  by  Homan  et  al.  (2001).  Not 
only  does  this  study  support  the  identified  problems,  it  also  postulates  that,  based  on 
McManus’s  VSM  manual  (2005)  and  through  MIT’s  Variation  Simulator,  bottlenecks  are 
caused  by  effects  external  to  the  process.  The  variability  simulation  uncovered  the  heavy 
workload  and  firefighting  support  that  the  AFRL  T&E  Office  displayed  when  PMs  failed 
to  follow  standardized  guidelines.  This  led  to  firefighting  from  the  AFRL  T&E  Office 
which  ultimately  led  to  the  bottleneck  at  AFFTC  when  more  than  enough  tests  plans  were 
compiled  at  the  FOA  for  approval. 

4.3  Acceptable  Cost  and  Optimal  Risk 

In  this  portion  of  the  analysis,  a  notional  optimal  level  of  risk  is  created  to 
evaluate  the  value  of  each  risk  factor  before  reaching  an  optimal  level  of  risk  and  the 
deciding  cost  in  flight  testing  them.  Risk  is  usually  expressed  in  terms  of  probability  of 
occurrence  and  impact,  which  leads  to  dollars  and  time,  and  each  project  will  have  to 
control  its  own  destiny  through  factoring  the  different  indications  of  risks.  SUAV  flight 
mishaps  may  be  all  too  common  for  PMs  who  have  flight  tested  them  and  those  who  have 
the  experience  to  explain  why  flight  mishaps  happen.  As  discussed  earlier  in  the  accident 
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rates  of  SUAVs,  it  is  all  too  familiar  to  the  DoD  and  the  FAA.  Therefore,  they  require  a 
thorough  review  in  flight  personnel  poliey,  training,  and  operations  for  SUAV  tests. 
Figure  4.4  shows  the  balaneing  between  the  level  of  risks  and  the  eosts  of  a  SUAV 
program  to  the  optimal  level  of  risk,  as  discussed  in  Chapter  3. 


Figure  4.4:  Acceptable  and  Optimal  Risk 
(Morgan,  1981) 


4.3.1  Perceptual  Factors  in  Assessing  Risk 

From  the  previous  analyses  in  cost  measurement  and  variability,  the  cost  of  risk 
reduction  at  the  AFFTC  safety  reviews  have  shown  to  be  minimal  compared  to  the 
overall  costs  of  the  entire  program,  suggesting  that  more  reviews  should  be  done  to  quell 
the  increasing  number  of  flight  mishaps.  Additionally,  the  time  it  took  for  additional 
reviews  at  AFFTC  was  less  than  anticipated,  but  that  was  because  of  recent  personnel 
changes.  Furthermore,  personnel  at  AFFTC  were  experiencing  variations  in  approving 
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low  risk  test  plans  when  too  many  programs  were  pushed  to  the  FOA  for  approval, 
causing  major  bottlenecks  at  AFFTC. 

In  light  of  the  two  previous  analyses,  why  would  the  PMs  and  the  AFRL  T&E 
Office  want  to  de-scope  the  level  of  oversight  from  AFFTC  on  low  risk  programs  given 
that  the  cost  and  time  required  for  AFFTC  to  review  them  were  not  as  high  as  originally 
thought?  In  other  words,  what  is  the  reason  why  people  are  reluctant  to  perform  risk 
analysis  (i.e.,  mishap  mitigations  at  AFFTC)  for  low  risk  programs  when  the  process  time 
is  short  and  the  cost  is  inexpensive?  The  key  issue  is  one  of  perception.  Cost 
measurement  results  show  that  the  AFFTC  processing  time  is  organizational  and 
personality-dependent — that  personnel  changes  in  the  future  may  not  have  the  same  test 
plan  approval  rate,  thus  adding  the  uncertainty  in  the  overall  process.  Additionally,  the 
PMs  and  the  AFRL  T&E  Office  felt  that  there  was  no  value  added  at  the  AFFTC  review 
process. 

So  why  do  SUAV  programs  have  to  spend  a  few  thousand  dollars  and  an  extra 
week  to  get  the  approval  from  AFFTC  when  there  is  no  perceived  value  added  from  the 
AFFTC  review  process?  The  answer  lies  in  the  optimal  level  of  risk  that  the  PM  is 
willing  to  take.  This  optimal  level  of  risk  taking  is  non-quantitative  and  must  be  based  on 
the  perceptual  factors  that  are  relevant  to  the  assessment  of  risk,  as  identified  by  Hillson 
and  Hulett  (2004). 

1.  Familiarity.'  The  extent  to  which  an  individual,  team,  or  organization  has 
previously  encountered  the  situation  drives  whether  the  risk  probability  is 
perceived  as  high  or  low.  Where  there  is  little  or  no  previous  relevant 
experience,  skill  or  knowledge,  the  degree  of  uncertainty  is  perceived  as 
higher  than  that  case  when  it  is  assessed  by  individuals  or  groups  who 
have  come  across  the  situation  before. 
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2.  Manageability.  The  degree  of  control  or  choice  that  can  be  exercised  in  a 
given  situation  drives  the  assessment  of  uncertainty,  even  if  the  perception 
is  illusory.  Where  a  risk  is  seen  as  susceptible  to  control,  risk  probability 
is  assessed  as  lower  than  in  situations  where  controllability  or  choice  are 
absent  (or  perceived  to  be  so). 

3.  Proximity.  If  the  possible  occurrence  of  a  risk  is  close  in  time  or  space  to 
those  assessing  its  probability,  it  will  be  seen  as  more  likely  than  risks 
which  might  occur  later  in  time  or  further  away  in  space. 

4.  Propinquity.  This  term  is  used  to  describe  the  perceived  potential  for  the 
consequences  of  a  risk  to  affect  the  individual  or  group  directly.  The 
closer  the  impact  is  to  those  assessing  the  risk,  the  higher  its  perceived 
probability  is. 


These  factors  are  hard  to  recognize  and  address,  as  they  and  other  perceptual 
influences  operate  subconsciously  when  individuals  and  teams  assess  risk  probability  and 
category  (Hillson  &  Hulett,  2004).  Another  critical  reason  why  low  risk  programs  have  to 
go  through  AFFTC  approval  review  is  to  hold  accountability  and  responsibility  in  the 
program;  these  “ways  of  seeing  and  understandings  ”  of  risk  in  organizational  settings 
were  seen  as  methods  of  deferring  blame  to  any  single  organization  with  respect  to  issues 
of  risk  and  responsibility  (Hutter  &  Power  2005).  After  all,  organizations’  and 
individuals’  interests  may  vary,  but  they  address  issues  in  test  flights  through 
considerations  of  commonalties  in  risk  management. 


4.3.2  Heuristics  and  Biases  in  Risk  Assessment 

In  assessing  risk,  other  factors  that  influence  the  PMs’  decisions  in  the  perception 
of  risk  are  known  as  heuristics  or  rules-of- thumb.  Hillson  and  Hulett  (2004)  define 
heuristics  as  internal  frames  of  reference  used  by  individuals  and  groups  to  inform 
judgment  when  no  firm  data  are  available.  Hillson  and  Hulett  (2004)  emphasize  the 
influence  of  introducing  two  types  of  heuristic  bias  into  assessments  in  situations  of 
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uncertainty.  The  first  type  of  bias  is  motivational  bias,  where  the  assessor  seeks  to 
improve  the  apparent  position  of  the  situation  by  modifying  the  estimate  of  risk 
probability.  The  second  type  of  bias  is  cognitive  bias,  which  arises  from  unconscious 
attempts  to  rationalize  the  lack  of  certain  knowledge  (Hillson  &  Hulett,  2004).  Of  these 
two  biases,  it  is  observed  that  motivational  bias  is  perhaps  more  commonly  found  in  PMs. 
The  author  also  noticed  in  the  various  interviews  that  PMs  were  more  likely  to  exhibit 
motivational  bias.  This  is  a  purely  anecdotal  impression  from  the  author  and  this  has  no 
hard  data  to  quantify  this  effect. 

Motivational  bias  is  difficult  to  identify  and  manage  as  the  person  or  organization 
assessing  the  risk  has  an  interest  in  influencing  the  results  of  the  analysis  (Hillson  & 
Hulett,  2004).  SUAVs  are  distinct  by  their  sizes,  costs,  and  complexity.  These  factors 
often  define  the  risk  of  their  test  flights.  The  presumption  of  motivational  bias  usually 
makes  the  probability  of  the  risk  of  SUAVs  seem  to  be  smaller  than  what  it  really  is, 
especially  in  order  to  reduce  the  perception  of  risk  among  key  stakeholders.  It  is  observed 
that  PMs  will  use  the  notion  of  their  SUAVs  as  being  small,  lightweight,  and  inexpensive 
to  de-value  the  purpose  of  any  extra  safety  reviews  that  may  cause  their  programs  to 
incur  more  costs  and  time.  Granted,  it  is  true  from  the  data  gathered  that  most  SUAV 
programs  were  deemed  as  low  risk,  but  the  cost  sensitivity  in  the  payloads  carried  by 
SUAVs  were  often  the  deciding  factor  in  categorizing  risks. 

4.3.3  Notional  Optimal  Level  of  Risk 

From  these  perceptual  and  heuristics  factors,  PMs  will  argue  that  the  absence  and 
noninvolvement  from  AFFTC  may  create  a  reduction  in  the  cost  of  risk  abatement,  which 
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reduces  the  total  costs  and  time  incurred  from  the  flight  approval  process  and  SUAV 
operation.  This  reduction  will  create  a  new  cost  of  risk  abatement  and  a  new  optimal  level 
of  risk,  as  shown  in  Figure  4.5. 


Figure  4.5:  Notional  Optimal  Level  of  Risk  (without  AFFTC  involvement) 


This  noninvolvement  from  AFFTC  in  low  risk  programs  will  be  manifested  by 
AFFTC  giving  full  approval  authority  on  low  risk  tests  to  the  AFRL  T&E  Office.  Not 
only  would  this  new  level  of  risk  abatement  reduce  the  time  and  costs  for  the  PMs  at 
AFRL,  but  it  would  also  give  the  reviewers  at  AFFTC  more  time  to  perform  their  main 
duties  at  AFFTC,  given  that  the  FOA’s  main  duties  come  from  the  flight  test 
organizations  at  AFFTC.  Furthermore,  since  90%  of  the  tests  coming  from  AFRL  are  low 
risk,  it  would  assume  that  the  costs  are  low  and  the  risks  can  be  negated.  Additionally,  the 
data  variability  analyses  have  shown  a  bottleneck  at  AFFTC  when  the  FOA  experiences 
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variation  in  approving  SUAV  test  plans  coming  from  AFRL.  This  new  optimal  level  of 
risk  results  in  a  lesser  total  cost,  and  its  expected  losses  from  risk  is  even  less  due  the 
overall  factors  from  the  PM’s  presumption  in  the  motivational  bias  and  heuristic 
approaches.  This  notional  optimal  level  of  risk  will  be  carried  to  the  latter  section  of  this 
chapter  to  create  a  notional  common  threshold  and  decision  rule  matrix. 

Noninvolvement  from  AFFTC  would  certainly  increase  the  number  of  SUAV 
programs  and  increase  R&D  in  SUAVs  and  their  systems.  However,  risks  are  still 
inherently  apparent  to  the  PMs  and  the  flight  testers.  Every  project  is  unique  and  SUAV 
projects  are  no  different  when  it  comes  to  safety.  The  mishap  rate  is  still  high,  and  the 
technologies  to  improve  SUAV  safety  records  are  still  being  developed;  therefore,  it  is 
prudent  to  approach  testing  with  attentiveness  when  no  AFFTC  involvement  is  granted. 

4.3.4  A  What-if  Scenario 

The  problems  of  unintended  consequences  are  high  whether  the  program  is 
categorized  as  low  risk  or  not.  Flight  testers  are  fully  aware  that  most  of  the  accidents 
happen  between  test  points  or  on  low  risk  activities  (Ward  et  ah,  2006).  Rules  and 
regulations  are  in  place  for  a  reason,  but  in  times  of  budget  crunches,  there  is  always 
pressure  to  eliminate  unnecessary  work  and  waste  when  results  confirm  non-added  value 
work.  However,  when  it  comes  to  proper  leadership  (AFMC/A3,  2006),  the  thought  was 
not  in  terms  of  what  the  testers  in  the  T&E  community  characterize  as  low,  medium,  or 
high  risk,  but  the  thought  stemmed  from  accountability  and  responsibility  terms — that  all 
flying  incurs  some  level  of  risk  and  proper  supervision  is  better  to  measure  and  mitigate 
risks.  In  an  evocative  scenario,  it  would  be  extremely  uncomfortable  and  an 
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embarrassment  for  the  test  team  to  explain  to  leadership  about  how  a  low  risk  mission 
flown  by  an  AFRL  contractor  experienced  a  Class  A  mishap  when  the  mishap 
investigation  revealed  that  no  rated  Operations  Group  Commander  (0-6)  oversight  was 
discussed,  as  dictated  by  the  MOA  between  AFRL  and  AFFTC.  This  thought-vignette 
was  one  of  the  arguments  against  an  improvement  initiative  from  both  AFRL  and  AFFTC 
to  reduce  the  scope  of  AFFTC  oversight  on  low  risk  programs,  and  was  quickly  rejected 
at  AFMC/A3.  It  is  the  very  same  reason  why  the  MOA  was  put  in  place  at  the  begirming 
of  the  joint  AFRL/ AFFTC  flight  safety  reviews. 

4.3. 5  Managing  Risk  Results 

One  of  the  challenging  outcomes  of  risk  assessment  is  that  it  is  difficult  to 
objectively  quantify  risks.  Hence,  the  qualitative  results  discussed  two  main  objectives. 
Two  questions  were  answered  in  the  optimal  risk  analysis  section:  Why  is  there  a  drive  to 
de-scope  the  AFFTC  oversight  of  low  risk  programs  when  AFFTC  time  and  cost  to 
review  were  short  and  less  expensive  than  originally  thought?  And  why  do  SUAV 
programs  have  to  spend  a  few  thousand  dollars  and  the  extra  week  to  get  the  approval 
from  AFFTC  when  there  is  no  value  added  to  the  program?  This  research  has  also 
included  the  top  level  acceptable  cost  of  optimal  risk  analysis  in  which  the  perceptual 
factors  and  heuristic  factors  were  taken  into  account  to  determine  whether  or  not  the  extra 
reviews  at  AFFTC  is  necessary.  Motivational  bias  was  seen  as  a  tool  that  most  PMs 
would  use  to  assess  risks  on  their  programs  based  on  the  notion  that  their  SUAVs  were 
small,  inexpensive,  and  can  be  replaced  with  commercial-off-the-shelf  parts.  The  results 
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from  cost  measurement  analysis,  variability  analysis,  and  optimal  risk  analysis  will  be 
tied  together  to  eome  up  with  a  threshold  and  reaeh  a  decision  rule  in  the  next  section. 

4.4  AFFTC  Threshold  and  Decision  Rule 

This  part  of  the  ehapter  will  sum  the  previous  three  analyses  and  determine  a 
reeommendation  for  an  acceptable  threshold  whieh  would  help  PMs  at  test  organizations 
reach  a  decision  rule  on  a  flight  test  approval  process  for  SUAVs.  According  to  Hillson 
(2004),  an  acceptable  threshold  defines  a  target  against  whieh  the  effectiveness  of 
responses  ean  be  measured.  Without  such  a  target,  too  much  effort  might  be  spent  on 
mitigating  risk  beyond  what  would  be  acceptable,  or  reviews  might  not  go  far  enough  in 
redueing  threats  and  enhancing  opportunities  (Hillson,  2004). 

Several  considerations  and  assumptions  were  made  when  attempting  to  estimate  a 
notional  cost  balance  statement  for  SUAVs.  From  previous  estimates,  the  total  SUAV 
assets  and  liabilities  came  to  about  $191,500  for  a  low  risk  program.  This  is  roughly  the 
same  as  a  Class  C  mishap  definition,  which  is  the  total  eost  of  property  damage, 
ineluding  all  aireraft  damage  between  $10,000  and  $200,000.  In  addition  to  monetary 
loss.  Class  C  mishaps  also  involve  injuries  that  result  in  five  or  more  workdays  lost. 
Beeause  of  the  approximate  SUAV  eost  assets  and  liabilities,  it  is  reeommended  that  the 
decision  rule  threshold  for  cost  remain  at  $200,000.  The  justification  behind  this  cost 
threshold  is  that  it  holds  true  to  the  average  eost  of  the  SUAV  flight  test  and  anything  less 
than  $200,000  in  total  assets  and  liabilities  is  deemed  a  Class  C  mishap  should  an 
accident  happen.  In  addition  to  the  cost  threshold,  there  is  the  size  and  weight  of  the 
SUAV’s  airframe  to  eonsider  should  it  erash  to  the  ground  and  eause  injury  and  damage 
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to  people  and  property.  A  notional  threshold  in  SUAV  size  and  weight  will  also 
determine  whether  the  program  should  go  to  the  extra  review  at  AFFTC. 

It  is  unknown  if  the  DoD  or  the  FAA  have  rules  and  regulations  for  SUAVs  based 
on  their  size  and  kinetic  energy.  However,  research  from  other  non-military  agencies, 
such  as  the  JAA  and  EUROCONTROL  (2004),  has  proposed  a  cut-off  line  with  respect 
to  SUAV  size  and  kinetic  energy  at  impact.  The  JAA/EUROCONTROL  reviewed  the 
worldwide  UAV  fleet  and  judged  the  capability  of  existing  SUAV  models  on  their 
workload,  masses,  kinetic  energy,  airspeed,  and  UAV  operator’s  ability.  The  report 
proposed  a  guideline  for  the  regulation  of  SUAV  systems.  Generally,  most  UAVs  less 
than  150  kg  are  classified  as  light  or  SUAV  systems  (JAA,  Weibel  &  Hansman,  2004). 
These  SUAVs  do  not  have  to  comply  with  the  same  regulations  as  manned  aircraft,  yet 
do  not  qualify  as  model  aircraft  in  Europe.  However,  they  are  required  to  operate  within 
restrictions,  such  as  daylight  operations  and  within  the  visual  range  of  the  operator,  and  in 
restricted  airspace  (JAA,  2004).  Larger  UAVs  weighing  more  than  150  kg,  with  a  kinetic 
energy  at  impact  of  more  than  95  kJ  flying  in  unrestricted  airspace  or  outside  the  visual 
range  of  the  operator  will  be  required  to  be  flight  tested  and  approved  in  a  similar  manner 
to  that  of  manned  aircraft  (JAA,  2004). 

4.4.1  Notional  Threshold 

Based  off  of  the  notional  cost  threshold  and  the  proposal  from  the 
JAA/EUROCONTROL  report,  this  research  proposes  using  the  same  standard  and 
regulation  based  on  the  size  and  energy  criteria  as  that  of  the  JAA/EUROCONTROL,  and 
recommends  that  the  DoD  and  FAA  adopt  the  same  technique,  or  something  similar. 


71 


Figure  4.6  provides  a  notional  decision  rule  matrix  incorporating  size  and  energy  criteria 
as  it  pertains  to  the  evaluation  of  SUAV  flight  test  approval  process.  The  notional 
decision  rule  matrix  is  flexible  and  can  be  used  given  the  dependencies  of  many  heuristic 
factors  based  on  risk  assessment  and  the  experience  that  comes  with  it.  These  factors 
include  the  environment  in  which  the  SUAV  is  flown,  the  maturity  of  the  system,  and  the 
product  weight,  size,  speed,  and  complexity.  The  decision  rule  matrix  assumes  that  the 
SUAVs  are  flown  in  restricted  airspace. 
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Figure  4.6:  Notional  Threshold 


The  framework  for  adoption  of  this  deeision  matrix  is  based  on  the  faet  that  low 
eost  SUAVs  with  simplistie  designs  should  require  less  review  time.  The  framework  also 
allows  the  test  team  and  those  at  AFFTC  and  AFRL  to  assess  the  risks  involved  in  some 
of  these  SUAV  programs  before  a  deeision  is  reaehed.  It  provides  test  teams  a  notional 
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“tipping  point”  where  AFFTC  safety  reviewers  would  essentially  add  value  to  the  flight 
approval  process.  For  example,  the  FOA  may  spend  a  greater  amount  of  time  on 
programs  that  are  high  cost  and/or  the  systems  involved  are  larger  than  the  average  sized 
UAV  that  is  usually  tested. 

4.4.2  Notional  Decision  Rule  Matrix  and  Outcome 

Sometimes,  decision-making  follows  a  simple  rule;  people  view  the  problem  in  a 
stereotyped  way  and  often  determine  if  the  given  situation  is  common  or  familiar  based 
on  their  past  experiences  (Klein,  1998).  Likewise,  if  people  decide  that  the  situation  is 
uncommon  or  unfamiliar,  then  they  follow  an  alternate  safer  decision  (Klein,  1998). 
Quadrants  1  and  4  in  the  decision  rule  matrix  clearly  demonstrates  Klein’s  successful 
decision  making  model  (1998),  that  the  situation  is  either  common  (from  past 
experiences)  or  uncommon  (alternate  safe  decision).  Quadrants  2  and  3  differ  in  their 
level  of  consequences,  as  the  decision  is  to  have  either  the  extra  review  or  no  extra 
review.  Quadrant  2  is  deemed  acceptable  in  daily  operations  because  of  its  conservative 
decision  which  leads  to  low  consequence  (Klein,  1998).  Quadrant  3  is  not  acceptable 
because  it  may  lead  to  highly  consequential  damage  or  injury  (Klein,  1998).  Figure  4.7 
shows  how  decisions  are  made  based  on  the  PM’s  experience  with  the  program  and  the 
given  situation. 
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What  We  Think  the  Situation  is 
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Figure  4.7:  Notional  Decision  Rule  Matrix  and  Outcome 
(Modified  from  Klein,  1998) 


4.4.3  Meta-language  and  Heuristics 

In  order  to  better  define  the  notional  decision  rule  quadrant  as  shown  in  Figure  4.6 
and  Figure  4.7,  one  may  be  inclined  to  describe  risk  using  meta-language.  Risk  meta¬ 
language  is  a  structured  description  of  a  risk  which  separates  cause,  risk,  and  effect. 
Meta-language  allows  risk  analysis  to  provide  a  three  part  structured  risk  statement  that 
should  ensure  risk  identification  through  a  heuristic  approach  (Hillson,  2008).  The  risk 
meta- language  format  can  be  written  like  this:  “Because  of  \cause(s)'\,  \risk'\  might  occur, 
which  would  lead  to  \effect(s)'\." 

Causes  are  present  facts  or  conditions  which  can  give  rise  to  risks  (Hillson,  2008). 
Once  the  cause  is  identified,  one  can  determine  the  risk  by  asking  “so  what?”  which 
would  lead  to  a  decision  on  the  uncertainties  about  the  cause.  The  effects  stem  from  the 
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conditional  future  state  that  the  risk  decision  produces.  With  respeet  to  the  notional 
decision  rule  quadrant,  the  risk  meta-language  can  be  written  in  four  separate  formats  for 
each  quadrant,  as  shown  in  Table  4.3. 


Table  4.3:  Meta-Language  and  Decision  Rule 


1.  “Because  my  UA  V program  is  expensive  and  its  impact  energy  is  high  [eause],  / 
should  increase  my  time  and  resources  in  risk  reduction  [risk  deeision],  which  should 
lead  to  low  mishap  probability  and  no  losses  [effeet].  ”  This  meta-language  implies 
the  UAV  program  is  expensive  and  heavy  at  impaet,  to  which  the  PM  would  make  the 
decision  to  have  the  extra  review  at  AFFTC  in  order  to  mitigate  future  risks. 

2.  “Because  my  UA  V program  is  expensive  and  its  impact  energy  is  low  [cause],  / 
should  make  some  effort  in  risk  reduction  [risk  decision],  which  should  lead  to  low 
mishap  probability  and  expect  low  losses  [effect].  "  This  meta-language  implies  that 
the  UAV,  although  lightweight,  is  still  expensive,  whieh  would  lead  the  PM  to  invest 
a  eonservative  amount  in  risk  reduction  at  AFFTC  in  order  to  reduce  future  mishaps, 
which  would  otherwise  cause  the  expensive  UAV  to  crash  and  incur  greater  losses. 
The  effect  is  a  lower  expeeted  loss  due  to  the  initial  risk  reduction  effort. 

3.  “Because  my  UA  V program  is  inexpensive  but  its  impact  energy  is  high  [eause],  I 
should  not  invest  too  much  time  and  effort  in  risk  reduction  [risk  deeision],  which 
may  yield  high  mishap  probability  from  risk  [effect].  ”  This  meta-language  suggests 
that  the  UAV,  although  heavy,  is  inexpensive  and  probably  simple  to  build.  The  risk 
decision  is  to  forgo  with  the  extra  review  due  to  its  inexpensiveness;  however,  the  PM 
should  expect  the  risk  of  failure  (mishaps)  to  be  high  sinee  it  does  not  go  through  the 
extra  review. 

4.  “Because  my  UA  V program  is  inexpensive  and  its  impact  energy  is  low  [cause],  / 
should  not  invest  too  much  time  and  effort  in  risk  reduction  [risk  deeision],  which 
could  lead  to  medium  or  low  probability  of  a  mishap  [effeet].  ”  This  meta-language 
suggests  that  the  UAV  is  inexpensive  and  lightweight,  therefore,  the  PM  should  not 
need  the  extra  review  before  flying,  however,  the  probability  of  a  mishap  is  minimum 
or  low  due  to  the  laek  of  reviews. 


An  area  of  eoncem  in  the  deeision  rule  is  where  SUAVs  exeeed  the  proposed 
mass  and  kinetic  energy  at  impaet  criteria  but  they  are  under  the  $200,000  threshold 
(deeision  quadrant  3).  The  big  mass  and  high  kinetie  energy  at  impaet  may  present  a 
hazardous  flying  experienee  should  a  mishap  oecur  on  the  ground  with  injury  to 
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personnel  involved.  This  would  create  a  cautious  choice  for  all  personnel  involved  in 
deciding  whether  or  not  AFFTC  is  needed  and  if  it  would  add  any  significant  safety 
comments  in  the  safety  review  process.  Therefore,  this  framework  and  decision  rule  is 
based  on  an  experienced  judgment  and  heuristic  ability  of  the  UAV  operator  and  the  test 
team  to  control  the  vehicle. 

In  summary,  risk  meta-language  is  a  useful  tool  for  clarifying  risk  descriptions. 
The  simple  three-part  framework  and  decision  rule  offer  general  guidelines  and  rule-of- 
thumb  principles  that  an  experienced  PM  may  be  able  to  find  risk,  uncertainties,  and  the 
effects  of  risk  decisions.  There  is  no  best  approach  in  making  a  decision  in  whether  or  not 
to  add  the  extra  review  from  AFFTC.  The  best  approach  may  be  answered  with  “it 
depends.”  It  depends  on  the  cost  of  the  UAV,  the  level  of  sophisticated  hardware 
involved  in  the  UAV,  the  level  of  details  and  complexity  of  the  risk  process,  and  any 
external  factors  that  play  in  UAV  test  flights. 

4.5  Application  of  Notional  Threshold  and  Decision  Rule  Matrix 

After  collecting  SUAV  data  with  total  cost,  mass,  and  impact  energy  (see 
Appendix  C),  this  research  plots  16  AFRL  SUAV  programs  using  the  notional  threshold 
and  decision  rule  matrix  for  further  analysis,  as  shown  in  Figure  4.8.  Notice  that  the 
majority  of  the  SUAV  programs  from  2011  were  lightweight  (less  than  150  kg)  and  less 
than  $200,000.  As  seen  in  Figure  4.8,  one  SUAV  program  had  cost,  weight,  and  kinetic 
energy  which  exceeded  the  notional  threshold.  This  program  experienced  a  flight  mishap, 
which  was  probably  caused  by  the  vehicle’s  immaturity  as  data  showed  that  the  vehicle 
had  less  than  one  hour  of  total  flying  time.  Another  SUAV  (quadrant  2)  cost  more  than 
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$200,000,  but  its  mass  and  kinetic  energy  were  less  than  the  notional  threshold;  it  also 
experienced  a  mishap.  However,  unlike  the  SUAV  mishap  in  quadrant  1,  this  vehicle  had 
more  flying  time  and  was  considered  mature.  The  third  SUAV  mishap  (quadrant  4)  was 
under  the  cost  threshold  and  proposed  mass  and  kinetic  energy.  This  particular  vehicle 
was  considered  mature  with  four  sorties  and  over  nine  hours  of  total  flying  time. 
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Figure  4.8:  2011  SUAV  Flight  Tests 

Based  on  the  limited  data  and  the  three  mishaps  given,  the  above  figure  shows 
that  there  is  no  eorrelation  between  flight  mishaps  and  an  SUAV’s  eost  and  their  physieal 
sizes  and  energy.  Nor  there  is  any  eorrelation  between  system  maturity  (total  flying 
times)  and  flight  failures.  Figure  4.8  also  shows  that  the  majority  of  SUAVs  submitted  for 
flight  testing  are  less  than  $200,000  and  below  that  of  the  proposed  mass  and  kinetie 
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energy  threshold.  Furthermore,  AFRL  mishap  data  from  previous  years  shown  in  Figure 
4.9  reveals  that  the  majority  of  mishaps  were  deemed  as  Class  E  events,  which  are 
occurrences  that  do  not  meet  reportable  mishap  classification  criteria,  but  are  important  to 
investigate  and  report  for  mishap  prevention.  A  total  of  46  mishaps  were  Class  E  events, 
eight  were  considered  as  Class  C  mishaps,  and  only  one  resulted  in  a  Class  B  mishap 
from  2007  to  2012. 
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Figure  4.9:  AFRL  Mishaps  by  Fiscal  Year 


Essentially,  SUAVs  are  sometimes  prone  to  system  failure;  therefore,  the  flight 
approval  process  for  SUAVs  should  be  based  on  the  judgmental  cost  and  the  proposed 
mass  and  impact  energy.  After  all,  SUAVs  are  still  a  relatively  new  platform  with  many 
opportunities  for  research  and  development.  An  appealing  aspect  of  SUAV  development 
is  the  fact  that  the  majority  of  AFRL  SUAVs  are  inexpensive,  simple,  and  tiny  to  build 
and  assemble  versus  larger  UAVs  or  manned  aircraft. 
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With  a  total  cost  less  than  or  equal  to  $200,000  and  a  small  and  light  weight 
airframe,  an  experienced  SUAV  operator  could  fly  the  SUAV  without  experiencing  a 
mishap  or  cause  extensive  damage  to  people  and  property.  Therefore,  in  consideration  of 
an  SUAV  program,  it  should  not  necessarily  adhere  to  the  extra  scrutiny  that  takes  extra 
time  and  spends  additional  resources  of  the  flight  test  team.  Innovations  and 
breakthrough  technologies  should  not  be  hampered  by  policy  for  the  sake  of  a  $200,000 
or  less  mishap.  However,  for  programs  that  are  well  over  $200,000,  its  size  and 
complexity  make  them  a  primary  target  for  safety  reviews.  Therefore,  it  is  within  reason 
to  have  the  extra  reviews  and  operational  authority  to  approve  the  flying  machine. 

4.6  Summary 

This  chapter  has  presented  the  results  and  analysis  from  data  gathered  from  the 
flight  test  offices.  Starting  with  the  cost  measurement  analysis  and  the  notional  cost 
balance  statement  for  an  average  SUAV  program,  the  initial  results  revealed  that  the  total 
time  and  cost  in  AFFTC  review  did  not  present  a  significant  investment  in  time  and 
money  from  the  test  team  at  AFRL.  Although  the  cost  and  time  were  minimal,  having  the 
additional  feedback  from  the  FOA  at  AFFTC  may  also  add  some  risk  mitigations  to  the 
test  plan.  However,  further  interviews  with  the  AFRL  T&E  Office  revealed  that  the 
feedback  coming  from  AFFTC  often  lacks  interest  and  value  to  the  test  plan. 
Additionally,  cost  sensitivity  analysis  results  confirmed  that  SUAV  program  costs  are  not 
sensitive  to  system  maturity,  but  rather  to  the  payloads  carried  by  the  SUAVs.  Data  from 
the  variability  analysis  confirmed  that  there  was  heavy  workload  and  firefighting  support 
at  the  AFRL  T&E  Office  when  PMs  and  test  teams  failed  to  submit  standardized  test 
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plans  on  time  with  adequate  notifications  of  upcoming  tests.  Variability  simulation 
showed  a  bottleneck  at  the  AFFTC  support  due  to  the  push  effect  caused  by  accumulated 
tests  and  late  notifications  from  the  AFRL  T&E  Office.  Lastly,  a  top  level  risk  analysis 
was  conducted  using  perceptual  factors  and  heuristic  factors  to  determine  whether  or  not 
the  extra  reviews  at  AFFTC  were  necessary.  A  notional  optimal  level  of  risk,  including 
the  noninvolvement  from  AFFTC,  which  decreased  the  total  costs  and  time  incurred  in 
the  flight  approval  process  and  SUAV  operation  was  developed,  but  includes  a  cautious 
warning. 

Finally,  in  combining  all  three  earlier  analyses,  this  study  creates  a  top-level 
AFFTC  threshold  and  decision  rule  recommendation,  which  was  determined  from  the 
average  $200,000  SUAV  program  cost  and  the  weight  and  impact  energy  of  the  SUAV. 
The  decision  rule  quadrant  and  the  successful  decision  model  should  allow  the  test  team 
and  those  at  AFFTC  and  AFRL  to  assess  the  risks  involved  in  SUAV  programs  before  a 
decision  is  reached.  It  also  allows  test  teams  to  determine  a  “tipping  point”  where  AFFTC 
safety  reviewers  would  essentially  add  value  to  the  overall  flight  approval  process. 
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V.  Discussions  and  Conclusions 


This  chapter  provides  discussions  and  conclusions  based  on  results  and  analysis 
of  this  research.  Recommendations  are  provided  with  a  threshold  and  decision  rule 
approach  in  the  critical  chain  of  SUAV  flight  approval  process.  Meta-language  and 
heuristics  are  used  to  identify  and  describe  risks  by  separating  the  causes  from  uncertain 
events.  The  following  section  summarizes  recommendations  for  SUAVs’  flight  approval 
process  and  their  typical  characteristic  attributes  which  include  being  inexpensive, 
simple,  and  often  fast  to  acquire.  Next,  this  chapter  wraps  up  with  the  research  objectives 
and  answers  the  questions  that  were  asked  in  the  introductory  chapter.  Additional 
observations  are  made  on  risk  management  by  leadership  and  the  lack  of  buy-in  from 
leadership  to  allow  AFRL  to  have  full  authority  on  low  risk  programs.  Lastly,  this  chapter 
summarizes  its  conclusions  with  limitations  for  this  study  and  suggestions  for  future 
research. 

5.1  Capturing  Research  Objectives 

This  study  provided  an  overarching  analysis  of  the  SUAV  flight  test  process 
currently  used  by  AFRL  with  oversight  from  AFFTC.  The  objective  was  to  establish  a 
measure  of  benefit  for  an  abbreviated  flight  test  process.  While  the  objective  has  been 
met  with  the  notional  decision  rule  matrix,  along  with  ranges  for  the  cut-off  cost  and 
SUAV  size  and  impact  energy.  However,  it  also  revealed  a  management  approach  that  is 
difficult  to  quantitatively  measure,  and  requires  years  of  program  management  and  flight 
test  experience  in  determining  the  risk  of  forgoing  AFFTC  involvement. 
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In  approaching  transparency  and  improving  safety  reviews  of  the  flight  test 
program,  multiple  layers  of  safety  reviews  add  benefits  to  mishap  mitigation  and 
accountability,  yet  they  may  be  time  consuming  for  PMs  who  want  to  accelerate  their 
tests.  Therefore,  is  it  necessary  for  SUAVs  to  have  the  same  criteria  applied  as  that  of  a 
larger,  manned  aircraft  in  flight  test  approval  process?  The  short  answer  is  yes  and  no. 
Safety  and  accountability  are  two  critical  issues  in  flight  testing;  however,  they  should 
not  create  an  environment  in  which  innovation  and  development  are  held  back  because  of 
a  simple  mishap  from  a  low  risk  test  plan.  This  leads  to  another  question  which  was 
raised  at  the  beginning  of  this  study:  Do  SUAVs  have  to  go  through  the  same  mishap 
investigation  and  reporting  process?  Evidence  from  AFRL  T&E  Office  has  suggested 
that  these  mishaps  were  minimal,  and  sometimes  they  were  expected.  There  is  no  doubt 
that  flight  safety  and  mishap  mitigations  are  applied  at  every  step  of  the  flight  approval 
process,  as  instructed  in  numerous  APIs  concerning  flight  testing.  Some  of  these  mishaps 
can  be  as  simple  as  a  crash  landing  on  an  unpaved  runway  that  caused  damage  to  the 
propeller  of  the  SUAV,  but  it  still  resulted  in  weeks  of  mishap  investigation  and  going 
through  the  return  to  flight  approval  process.  Nevertheless,  not  all  mishaps  are  minute;  a 
program  at  threshold  price  of  $200,000  with  expensive  payload  would  certainly  raise 
concerns  if  there  was  no  oversight  on  safety  reviews  from  an  appropriate  authority. 

In  contrast  to  the  traditional  military  standards  on  SUAV  flight  tests,  the  DoD 
ought  to  look  into  research  from  non-military  specifications.  This  study  has  developed  a 
notional  decision  rule  based  on  the  proposed  JAA/EUROCONTROL  (2004)  technique  on 
SUAV  weight  and  kinetic  energy  at  impact.  The  consequences  of  using  such  a  technique 
would  simplify  the  SUAV  flight  test  approval  process  and  could  save  money  in  the  long 
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ran.  With  a  simplified  approval  process,  R&D  organizations  might  be  able  to  increase 
and  expedite  the  weapons  development  needed  for  the  warfighters.  This  coincides  with 
the  current  DoD  policy  of  supporting  intelligence,  surveillance,  and  reconnaissance 
missions  globally.  Moreover,  an  abbreviated  approval  process  for  an  SUAV  is  greatly 
needed  for  rapid  reaction  and  rapid  deployment  to  the  theatre  of  operation  to  support  the 
warfighters,  as  opposed  to  the  generic  flight  test  approval  process. 

Additionally,  data  from  cost  sensitivity  and  variability  analyses  showed  that 
system  maturity  costs  were  varied  with  SUAV  payload  costs.  Program  managers  are 
always  fighting  system  instability  and  the  risk  of  uncertainties  in  the  program.  These 
uncertainties  may  be  caused  by  outside  influences  such  as  technology,  environment,  and 
political  and  financial  situations.  By  prioritizing  a  list  of  “must  haves”  and  “good  to 
haves,”  test  teams  may  be  able  to  minimize  the  cost  of  the  program  and  also  make  it  more 
manageable  in  determining  risks  and  use  the  decision  rale  to  determine  whether  to 
include  reviews  by  the  AFFTC  authority.  The  variability  analysis  showed  a  bottleneck 
occurring  at  the  FOA  due  to  variability  in  the  process  caused  by  the  constant  flooding  of 
test  packages  coming  from  AFRL  to  the  FOA. 

In  summary,  a  generic  standardized  regulation  for  SUAVs  may  help  the 
separation  of  risk  assessment  and  risk  management.  Differentiating  between  the  two  may 
prove  helpful  in  determining  risk  analysis  when  weighing  the  benefits  of  an  abbreviated 
flight  test  process  and  the  costs  associated  with  going  through  the  entire  flight  approval 
process.  Using  a  current  generic  standardized  regulation  in  flight  test  approval  process 
may  have  the  advantage  of  consistency,  predictability,  and  administrative  efficiency. 
However,  there  are  also  disadvantages  in  having  SUAVs  going  through  the  same 
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standardized  flight  test  approval  process  as  those  of  larger  manned  aircraft.  These 
disadvantages  can  include  overgeneralizing  SUAVs  as  having  a  complex  integrated 
system  like  those  of  larger  manned  aircraft.  Another  disadvantage  is  the  misallocation  of 
resources  and  schedule;  unlike  larger  manned  aircraft,  SUAVs  are  generally  operated  by 
a  small  team  of  engineers  and  observers.  Lastly,  by  putting  SUAVs  through  the  same 
flight  test  approval  process  required  of  larger  manned  aircraft,  there  will  be  some  delays 
which  may  eventually  cause  research  and  development  to  stop.  This  mixture  of  an  overly 
unwieldy  risk  assessment  policy  and  the  potential  for  interrupted  scientific  research 
would  probably  cause  delays  and  frustration  for  the  PM  and  the  test  team. 

5.2  Impact  of  Leadership  Risk  Management  Style 

One  of  the  major  surprises  from  this  study  was  the  unexpected  perspective  from 
AFRL  on  the  lack  of  value  and  attention  provided  by  AFFTC  reviews.  It  should  be  noted 
that  this  study  does  not  suggest  that  the  FOA  at  AFFTC  lacks  the  expertise  to  provide 
adequate  reviews  to  the  safety  of  the  test  missions.  This  study,  however,  advocates  the 
notion  of  a  “tipping  point”  where  AFFTC  safety  reviewers  would  essentially  add  value  to 
the  overall  flight  approval  process,  as  discussed  in  the  previous  chapter.  This  tipping 
point  is  determined  using  the  notional  decision  rule  matrix.  With  manning  levels  in  the 
Air  Force  already  strained,  and  the  fact  that  90%  of  AFRL’s  flight  test  programs  are 
considered  by  the  AFRL  and  AFFTC  as  low  risk,  the  level  of  authority  and 
empowerment  in  low  risk  programs  should  be  delegated  to  those  personnel  in  the  field. 
This  delegation  would  greatly  benefit  both  parties  if  low  risk  programs  were  approved  at 
AFRL  instead  of  AFFTC.  It  would  also  reduce  the  bottleneck  problem  at  the  FOA  while 
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increasing  opportunities  for  further  flight  testing  by  AFRL.  Homan  et  al.  (201 1)  also 
identified  these  issues  and  the  potential  opportunities  to  improve  the  flight  test  proeess. 

A  problem  here  lies  in  the  risk  adverse  mismanagement  style  of  leadership.  Stulz 
(2009)  and  Taleb  et  al.  (2009)  diseuss  ways  that  organizations  mismanage  risk;  one  of 
them  is  the  mistake  of  relying  on  historieal  data  and  beeoming  convinced  that  studying 
the  past  will  help  better  manage  risk  in  the  future.  In  essence,  leadership  relies  on  these 
historieal  events  (i.e.,  past  mishaps  from  AFRL  test  flights)  to  control  risk  by  predicting 
extreme  events.  However,  low-probability,  high-impaet  events  are  almost  impossible  to 
foreeast,  and  '^because  of  socioeconomic  randomness,  there  is  no  such  thing  as  a  ‘typical’ 
failure  or  a  ‘typical  success  (Taleb  et  al.,  2009).  Numerous  joint  AFFTC/AFRL 
proposals  have  been  fiinneled  up  to  leadership  for  the  delegation  of  low  risk  programs  to 
AFRL,  yet  their  status  is  unknown  or  leadership  refuses  to  consider  them.  This  overly 
eonservative  risk  management  approach  of  leadership  ereates  a  non-empowered 
environment  for  those  who  want  to  test  their  programs  and  for  those  who  review  them. 

5.3  Conclusions 

This  researeh  effort  sought  to  establish  top-level  quantitative  and  qualitative 
measures  of  benefit  for  an  abbreviated  flight  test  proeess  for  SUAVs.  Issues  such  as  cost 
sensitivity,  proeess  improvement,  and  risk  management  were  discussed  in  this  study.  The 
literature  review  exposed  performance  gaps,  cost  and  data  variability,  as  well  as  buy-in 
from  AFRL  and  AFFTC  leadership  to  delegate  low  risk  programs  to  AFRL;  however, 
this  was  rejeeted  due  to  the  laek  of  aeeountability  in  small,  low-risk  AFRL  programs.  The 
literature  also  reviewed  the  importanee  of  flight  test  and  the  faetors  that  affeet  the 
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effectiveness  of  the  flight  testing.  The  increasing  trends  in  SUAV  technology 
development  combined  with  the  ever  increasing  DoD  investment  for  SUAVs  in  the 
military  has  allowed  greater  R&D  effort,  yet  it  is  sometimes  limited  and  scaled  back 
because  of  outdated  policy  and  regulations  on  SUAV  operations  in  the  NAS  and 
restricted  airspace.  The  methodology  section  explained  the  research  design  for  this  study 
based  on  cost  and  time  criteria,  combined  with  data  variability  simulation  and  a  heuristic 
approach  to  risk  management. 

Research  and  development  in  new  or  un-tested  technology  often  times  is 
susceptible  to  additional  risks,  such  as  flight  mishaps  while  airborne  and  during  takeoff 
and  landing.  Often  times,  some  form  of  risks  in  new  R&D  technologies  may  be 
acceptable  in  the  pursuit  of  knowledge.  For  some  risks,  such  as  SUAV  flight  testing,  it 
produces  benefits  to  flyers  and  operators  in  the  form  of  greater  understanding  of  the 
vehicle’s  flight  characteristics  and  technological  improvements.  The  findings  from  this 
research  conclude  with  a  SUAV  threshold  and  decision  rule  matrix  that  will  empower 
leadership  to  make  decisions  based  on  relevant  data  appropriate  for  SUAVs. 

5.4  Limitations  and  Future  Research 

There  is  no  doubt  that  the  findings  from  this  research  are  limited  due  to  the 
assumptions  that  it  has  made.  These  limitations  stem  from  the  lack  of  adequate  data  in 
manpower  and  resources  necessary  to  capture  the  entire  SUAV  business.  A  larger  and 
broader  examination  of  each  SUAV  program  may  give  rise  to  improved  assumptions. 
Another  limitation  is  the  data  quality  in  this  study,  as  such  data  are  often  limited  in 
distribution  due  to  the  sensitivity  of  these  programs.  There  are  definitely  more  concerning 
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issues  than  what  is  captured  in  cost  and  SUAV  weights  and  sizes  in  the  flight  test 
approval  process.  Approving  routine  and  safe  flight  tests  for  SUAVs  involve  several 
issues  that  touch  on  nearly  every  aspect  of  the  aviation  field,  including  safety,  security, 
air  traffic,  regulatory,  and  socio-economic.  These  categories  should  be  addressed  and 
assessed  in  future  research  to  present  a  grander  view  of  the  factors  and  their  impacts 
associated  with  SUAV  flight  tests. 

Although  this  study  addresses  some  of  the  problems  and  shortcomings  from  both 
AFFTC  and  AFRL  organizations  and  also  provided  a  decision  rule  recommendation 
based  on  its  methodology,  there  are  still  areas  that  can  be  improved  and  expanded  upon. 
With  better  cost  data  from  all  previous  years’  of  AFRL  flight  test  programs,  future 
research  would  be  improved  on  SUAV  management  costs  and  their  estimates  for  further 
cost-benefit  analysis.  Another  area  of  research  that  would  greatly  benefit  the  flight 
approval  process  for  SUAVs  is  to  study  the  variation  of  UAV  sizes  and  costs  and 
determine  if  there  is  variability  in  the  processing  time  based  on  their  sizes  and 
manufacturing  and  operating  costs. 

This  research  had  difficulty  determining  the  optimal  level  of  risk  when 
approaching  the  approval  process  in  the  critical  chain.  So  the  optimal  level  of  risk  and  the 
new  total  cost  presented  in  this  study  is  merely  an  empirical  experiment  based  on 
available  data.  Future  research  should  utilize  the  notional  optimal  level  of  risk  and 
determine  the  appropriate  cost  of  risk  abatement  in  order  to  find  a  more  quantitative 
overall  total  cost  and  optimal  level  of  risk.  Future  research  should  also  conduct  more 
appropriate  interviews  with  the  FOA  at  AFFTC,  program  leaders,  and  SUAV  operators  in 
the  overall  process.  This  is  an  area  that  would  greatly  benefit  from  further  discussion  and 
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could  be  broadened  in  future  researeh  that  ineludes  SUAV  operators’  workload, 
situational  awareness,  and  human  effectiveness  issues  in  SUAV  operating  environments. 
Lastly,  since  this  research  only  focused  on  one  flight  test  approval  proeess  being  done  per 
the  MOA  between  AFFTC  and  AFRL,  it  is  strongly  recommended  that  future  follow-on 
research  should  seek  to  generalize  and  validate  this  approaeh  and  methodology. 
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Appendix  A:  Memorandum  of  Agreement  between  AFRL  and  AFFTC 
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MEMORANDUM  OF  AGREEMENT 
BETWEEN 

AIR  FORCE  RESEARCH  LABORATORY 
AND 

AIR  FORCE  FLIGHT  TEST  CENTER 

1 .  PURPOSE.  This  memorandum  of  agreement  is  to  establish  a  working  relationship 
and  processes  to  support  the  Air  Force  Flight  Test  Center  (AFFTC),  412*  Test  Wing, 
412*  Operations  Group  operational  control  (OPCON)  for  all  flight  activities  conducted 
by  the  Air  Force  Research  Laboratory  (AFRL).  The  AFRL  has  overall  responsibility  for 
program  management  and  administrative  support  for  its  research  programs.  The  AFFTC 
has  responsibility  for  the  safe  and  effective  conduct  of  flight  operations  for  AFRL 
research  programs  requiring  flight  activities.  The  AFMC/A3  defers  OPCON  to  the  412* 
Operations  Group  Commander  (OG/CC)  for  AFRL  research  programs  involving  military, 
government  civilian  or  contractor  flight  operations.  To  avoid  duplicate  reviews, 
inspections  or  evaluations,  an  integrated  team  approach  for  conducting  program  flight 
operation  oversight  will  be  used. 

2.  REFERENCES.  The  responsibilities  of  flying  operations  under  the  supervision  of 
the  AFFTC  and  the  412*  Operations  Group  are  detailed  in  Air  Force  Instructions. 

Ground  and  flight  activities  will  comply  with  applicable  regulatory  guidance. 
Modifications  of  and  exceptions  to  AFI  guidance  must  be  coordinated  through  the  412* 
OG/CC,  in  writing,  to  the  designated  Office  of  Primary  Responsibility  for  the  specific 
instruction.  Below  is  a  list  of  the  primary  Instructions  applicable  to  flight  operations. 

AFI  91-206(1) 

AFI 91-204  and  AFMC  Sup  1 
AFI  91-202  and  AFMC  Sup  1 
AFFTCI91-5 

AFI  99-103  and  AFMC,  AFRL  supplements 

AFI  1 1-401  and  AFMC  Sup  1 

AFI  1 1-202  volumes  1, 2,  and  3;  AFMC  supplements 

AFI  1 1-2FT  volumes  1,  2,  and  3 

AFJI  10-220  and  AFMC  Sup  1 

AFI  10-206 

3.  SCOPE.  This  agreement  shall  apply  to  military,  government  civilian  and  contractor 
flight  operations  regardless  of  program  size  or  location  of  actual  flight  operations. 

4.  RESPONSIBILITIES.  AFRL  will  continue  to  use  processes  in  place  for  managing 
its  test  programs  as  outlined  in  AFI  99-103  and  its  supplements.  The  AFFTC  and  412* 


89 


2006-06-06MA 


Operations  Group  will  provide  effective  flight  operations  oversight  of  all  AFRL  test 
planning  and  flight  execution.  As  clarification,  the  following  specifics  apply: 

4.1  AFRL  RESPONSIBILITIES. 

a.  Provide  the  41 2'*'  OG  with  program  procedures  and  documentation  to  meet  the 
intent  of  AFT  11-401  and  its  ATMC  and  AFFTC  supplements. 

1)  Monthly,  prior  to  the  last  day  of  any  month,  the  AFRL  will  provide  the 
412*  OG  with  an  electronic  summary  of  all  program  flight  activity 
accomplished  by  each  program  over  the  last  month,  and  the  scheduled 
flight  activity  expected  for  the  next  month. 

2)  As  a  minimum,  each  program  will  be  required  to  document  individual 
flight  authorizations  using  Aeronautical  Orders,  IMT  Form  81  or  IMT 
Form  5416  through  the  412*  OG.  This  will  be  accomplished  even  if  the 
aircraft  to  be  flown  is  not  USAF  owned. 

3)  Daily  flight  authorization  will  be  accomplished  locally  using  the 
procedures  detailed  in  AFI 1 1-401  and  its  AFMC  and  AFFTC 
supplements.  The  owning  AFRL  Division  Chief  will  act  as  the  flying 
squadron  commander  and  authorize  the  daily  flying  schedule,  with  copies 
faxed  to  the  412*  OG  before  flight. 

b.  Provide  the  412*  OG  with  appropriate  and  sufficient  evidence  of  aircrew  and 
operator  qualifications,  training  and  currencies  required  to  perform  in-flight  duties 
as  detailed  in  the  program  test  plan  and  to  meet  the  intent  of  AFI  1 1-202V1,  V2, 
V3  and  their  AFMC  and  AFFTC  supplements. 

1)  The  specific  approval  for  in-flight  duties  will  be  given  by  the  412*  OG. 
It  is  the  AFRL  program  manager’s  responsibility  for  ensuring  individuals 
performing  any  in-flight  duties  are  certified  as  qualified  and  current. 
Modifications  of  and  exceptions  to  AFI  guidance  must  be  coordinated 
through  the  412*  OG/CC,  in  writing,  to  the  designated  Office  of  Primary 
Responsibility  for  the  specific  instruction. 

2)  AFRL  programs  that  fly  with  airborne  and/or  ground  station  assets 
valued  at  over  $200,000,  is  assessed  as  MEDIUM  or  HIGH  risk  or  fly 
under  an  FAA  approved  Certificate  of  Authorization  (COA)  as  outlined  in 
FAA  regulation  7610.4  will  comply  with  all  AFI  10  and  1 1  series 
instructions. 
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c.  Provide  a  rated  billet  at  AFRL  for  the  AFFTC  to  fill  with  a  Major  or 
Lieutenant  Colonel  acting  as  a  local  Test  Representative  representing  the  412* 
Operations  Group.  Test  Representative  responsibilities  will  be  detailed  in  the 
position  job  description.  IXities  will  be  determined  collaboratively  with  AFRL 
and  the  4 12*  OG. 

d.  Coordinate  the  technical  and  safety  review  information  obtained  through 
standard  AFRL  Configuration  Control,  Technical  Review  and  Safety  Review 
Board  (CCByTRB/SRB)  processes  with  AFFTC  organizations. 

1)  For  all  AFRL  research  programs  involving  flight  operations,  copies  of 
CCB/TRB/SRB  packages  and  test  plans  will  be  provided  to  the  412* 
OG/CC  at  least  fourteen  days  prior  to  the  first  planned  test  activity.  More 
time  may  be  required  for  review  and  coordination  if  the  program  is 
assessed  by  AFRL  safety  as  MEDIUM  or  HIGH  risk. 

2)  AFFTC  I  91-5  details  the  signature  authority  for  LOW,  MEDIUM  and 
HIGH  risk  test  plans  for  AFFTC  programs  and  is  modified  below  to 
support  this  MO  A, 


ailGAXIZA.T[OKAL  LEVEL 

LOWSaSE 

ySEDiSVUmSK 

memmsK 

Ops  Group  CC  or  equivalent 

Approve 

Info 

[nfb 

Test  Wing  CC  ot  equivalent 

Coonl 

Approve 

Info 

AFFTC  CC 

Coord 

Coord 

Approve 

LOff^  risi  lest  plans  mil  be  approved  by  the  o-wning  AFRL  Division  Chief  or  their  Deputy 
and  coordinated  through  412'”  OG  before  the  start  of  any  flight  operations.  MEDIUM 
risk  test  plans  will  be  Jointly  approved  by  the  owning  AFRL  Technical  Directorate 
Director  and  the  4  if  TW/CC.  HIGH  risk  test  plans  will  be  Jointly  approved  by  the 
AFFTC/CC  and  the  AFRUCC.  A  formal  approval  briefing  may  be  requested  by  the 
approval  authority  for  any  test  plan,  but  is  required for  all  HIGH  risk  test  plans.  All 
coordination  comments  will  be  resolved  before  final  approfval  of  test  execution  and  made 
available  to  both  AFFTC  and  AFRL  leadership. 

3)  AFFTCI  91-5  details  the  signature  authority  for  LOW^  MEDIUM  and 
HIGH  risk  test  cards  for  AFFTC  programs  and  is  modified  to  support  this 
MO  A  as  follows: 

The  test  cards  must  be  approved  before  each  mission  but  no  earlier  than  one  week  before 
the  mission.  For  LOW  RISK  tests,  the  AFRL  Division  Chief  (or  equivalent)  will  approve 
the  cards.  For  MEDIUM  RISK  tests,  the  AFRL  Technical  Directorate  Director  and  the 
412^  Operations  Group  Commander  (or  equivalent),  will  approve  the  cards.  For  HIGH 
RISK  tests,  the  AFRL/CC  and  the  412^^  Test  Wing  Commander  (or  equivalent)  will 
approve  the  cards. 

Once  a  mission  is  briefed  and  the  mission  slips  due  to  a  non-scfety  related  reason,  but 
otherwise  remains  unchanged,  the  appropriate  approving  official  may  elect  to  grant 
approval  by  telephone  or  on  an  “until-flowW*  basis  for  up  to  one  week  after  original 
approval  If  more  than  one  weeA  has  passed  since  the  original  approval,  the  approval 
authority  must  be  contacted  and  the  date  on  the  signature  updated. 
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4)  Test  conduct  will  otherwise  comply  with  AFFTCI  91-5  procedures 
unless  modified  or  equivalent  procedures  are  approved  through 
AFFTC/SET  and  the  41 2“’  OG/CC. 

e.  Provide  a  Government  Flight  Representative  (GFR)  for  all  AFRL  programs 
that  fall  under  this  MOA  and  require  GFR  oversight.  The  assigned  GFR  will 
coordinate  with  both  the  AFRL  and  412*  OG  when  performing  their  duties. 

f.  Act  as  the  lead  agent  and  liaison  with  the  412*  OG  in  all  cases  where 
configuration  control  and  aircraft  modifications  affect  the  research  program’s  test 
plan  approved  baseline  configuration. 

4  2  AFFTC  RESPONSIBILITIES. 

a.  Provide  the  AFRL  with  a  flight  authority  review  and  approval  of  all  flight 
activities  to  ensure  AFRL  test  plans  are  conducted  in  accordance  with  applicable 
Federal  Aviation  Regulations  and/or  Air  Force  10  and  1 1  series  instructions  and 
their  AFMC  ^md  AFFTC  Supplements.  This  responsibility  may  be  delegated  to  a 
Government  Flight  Representative  (GFR)  when  required  by  contract. 

1 )  Process  all  waivers  to  governing  regulations  concerning  flight  test 
operations. 

2)  Assist  AFRL  with  establishing  and  approving  pilot  and  operator 
qualification,  training  and  currency  standards  for  AFRL  research  programs 
involving  flight  activities  to  meet  the  intent  of  AFI  1 1-202V1,  V2,  V3, 

AFI  1 1-2FTV1,  V2,  V3  and  their  AFMC  and  AFFTC  supplements. 

3)  Assist  AFRL  with  establishing  and  approving  flight  authorization 
requirements  for  AFRL  research  programs  involving  flight  activities  to 
meet  the  intent  of  AFI  11-401  and  its  AFMC  and  AFFTC  Supplements. 

4)  Assist  AFRL  with  establishing  and  executing  an  Operational  Risk 
Management  processes  associated  with  AFRL  research  programs. 

5)  Establish  processes  to  provide  AFRL  research  programs  involving 
flight  activities  with  the  Flight  Crew  Information  File  (FCIF). 

b.  412*  OG  will  provide  AFRL  aircraft  mishap  reporting  and  accountability 
assistance  as  the  flight  authority  for  all  flight  operations.  Reporting  and 
accountability  will  be  in  accordance  with  AFI  91-202, 91-204  and  91-206(1)  and 
AFMC  and  AFFTC  supplements. 
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1 )  412  OG  will  be  the  grounding  and  retum-to-fly  apj»x>val  authority  for 
individuals  involved  in  a  Class  A  or  Class  B  mishi. 

2)  AFRL  will  retain  mish^  accountability  and  investigation 
responsibilities  unless  transferred  lAW  AFI  guidance. 


the  technical  and  safety  review  information  obtained 
n  Control,  Technical  Review  and  Safety  Review 

Board  (CCBmiB/S^)  processes  with  AFFTC  organizations  to  support  AFRL 
^earch  programs  while  complying  with  the  intent  of  Air  Force  91  and  99  series 
Instructions  and  their  AFMC  and  AFFTC  supplements. 


1 )  Provide  safety  and  technical  assistance  for  AFRL  research  programs 
that  ^  outside  tite  expertise  of  AFRL.  This  may  include,  but  is  not 
limited  to,  providing  a  representative  &om  within  the  412**“  OG  to  ntti-pd 
program  planning  meetings,  assist  with  the  preparation  of  safety  and/or 
technical  documentation  or  attend  TRB  and/or  SRB  boards. 

2)  For  all  MEDIUM  and  HIGH  risk  assessed  test  plans,  a  412*  OG 
member  will  be  assigned  to  support  the  AFRL  research  program  with  test 
planning  and  integration  with  AFFTC  technical,  safety  and  flight  test 
execution  processes. 


5.  TERM  OF  AGREEMENT  AND  RIGHT  TO  TERMINATE.  This  memorandum 
bewm^  effective  on  the  date  signed  by  both  organizations  and  remains  in  efifect 

^  Tennin^on  of  this  agreement  is  only  valid  if  both  the  AFRL/CC  and  the 
412th  OG/CC  agree  in  writing  to  terminate  the  agreement  Amendments  to  this 
agre^ent  may  be  made  with  at  least  90-days  notification  by  ei&er  party,  in  writing 
detail^g  the  procedures  and  processes  to  be  amended.  This  agreement  shall  be  reviewed 


TED  F.  BOWLDS,  Maj  Gen 
Commander,  AFRL 


CURTIS  M.  BEftKE.  Maj  Gen 
Commander,  AFFTC 


D.  NEW,  Maj  Gen 
Btor  of  Operations 
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Appendix  B:  Current  Flight  Test  Process  VSM 
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Appendix  C:  Small  UAV  Costs  and  Mishap  Data 


SUAV 

Cost  Per  Vehicle 

Vehicles  Flown 

Total  Amount 

System  Maturity 

Sorties 

Hours 

Weight  (kg) 

Speed  (m/s) 

KE  (KJ) 

Mishaps 

Quadrant 

Alpha  60 

$ 

800.00 

7 

$ 

5,600.00 

Mature 

78 

20.33 

7.7 

24.7 

2.3 

4 

Alpha  60 

$ 

400.00 

1 

$ 

400.00 

Mature 

- 

- 

- 

- 

- 

- 

Arrow  MAV  Glider 

$ 

1,200.00 

1 

$ 

1,200.00 

Prototype 

6 

0.3 

0.9 

8.7 

0.0 

4 

Basset 

$ 

390,000.00 

1 

$ 

390,000.00 

Prototype 

3 

0.9 

317.5 

61.7 

604.3 

Yes 

1 

Bat-3 

$ 

318,000.00 

1 

$ 

318,000.00 

New 

18 

44 

14.5 

30.9 

6.9 

Yes 

2 

Big  Stik 

$ 

1,100.00 

5 

$ 

5,500.00 

New 

80 

23.3 

9 

31.4 

4.4 

4 

Blacktip 

$ 

6,200.00 

2 

$ 

12,400.00 

Prototype 

73 

14.85 

5.4 

49.9 

6.7 

4 

BVM  HiVast 

$ 

28,000.00 

1 

$ 

28,000.00 

New 

12 

1.5 

31.7 

102.9 

167.8 

3 

DG303 

$ 

17,800.00 

2 

$ 

35,600.00 

New 

23 

20.8 

- 

- 

- 

- 

Intimidator 

$ 

1,100.00 

5 

$ 

5,500.00 

New 

35 

12.9 

9 

44.8 

9.0 

4 

MOP  MAV  BDI 

$ 

24,000.00 

1 

$ 

24,000.00 

New 

2 

0.2 

4.5 

49.9 

5.6 

4 

Osprey  (UAV) 

$ 

2,333.00 

7 

$ 

16,331.00 

Mature 

40 

22.15 

27.2 

33.4 

15.2 

4 

Radian  RC 

$ 

250.00 

1 

$ 

250.00 

Mature 

7 

0.3 

0.9 

8.7 

0.0 

4 

Rascal 

$ 

549.00 

1 

$ 

549.00 

Mature 

31 

4.5 

- 

25.7 

- 

- 

RMAX 

$ 

380,000.00 

2 

$ 

760,000.00 

Mature 

0 

0 

93 

11.3 

5.9 

2 

ScanEagle 

$ 

100,000.00 

1 

$ 

100,000.00 

Mature 

1 

1 

20 

41.2 

17.0 

4 

Silver  Fox 

$ 

100,000.00 

1 

$ 

100,000.00 

Mature 

4 

9.5 

16.3 

28.3 

6.5 

Yes 

4 

Skate 

$ 

30,000.00 

4 

$ 

120,000.00 

New 

64 

24.5 

0.9 

25.7 

0.3 

4 

Tandem  Wing 

$ 

3,000.00 

2 

$ 

6,000.00 

Mature 

7 

0.6 

- 

14 

- 

Yes 

- 

Unicorn 

$ 

17,000.00 

1 

$ 

17,000.00 

Mature 

39 

15.9 

4.5 

24.7 

1.4 

4 

Unicorn 

$ 

22,400.00 

1 

$ 

22,400.00 

Mature 

- 

- 

- 

- 

- 

- 

Unicorn 

$ 

18,700.00 

3 

$ 

56,100.00 

Mature 

- 

- 

- 

- 

- 

- 

$ 

1,462,832.00 

$  2,024,830.00 

523 

218 
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SUAV  Program  Asset  Cost 

$133,000.00 

$  133,000.00 

Test  Execution  Cost 

$ 

30,000.00 

$ 

30,000.00 

AFRL  Review  Cost 

AFRL  Risk  Abatement  Cost 

$ 

2,054.79 

$ 

2,100.00 

Value  Added  Work 

$  20,000.00 

$ 

20,000.00 

Non-vaiue  Added  Work 

Waste 

$ 

1,178.08 

$ 

1,200.00 

Necessary  Waste 

$ 

1,534.25 

$ 

1,500.00 

AFFTC  Review  Cost 

AFRL  Risk  Abatement  Cost 

$ 

2,465.75 

$ 

2,500.00 

Value  Added  Work 

$ 

410.96 

$ 

400.00 

Non-value  Added  Work 

Waste 

$ 

821.92 

$ 

800.00 

Necessary  Waste 

$ 

- 

$ 

- 

Total  UAV  Asset  and  Liabilities 

$ 

191,465.75 

$ 

191,500.00 

$66,500  X  2  SUAVs 
Estimated  range  cost 

1  GS-15  with  5  working  days  ($150K  salary) 

1  GS-15  [$150K salary)  with  2  working  days  +  4  junior  member  [$70K salary)  with  25  working  days 

1  GS-15  [$150K salary)  with  1  working  day  +  4  junior  members  [$70K salary)  with  1  working  day 
4  junior  members  [$70K salary)  with  2  working  days 

1  0-5  ($150K  salary)  with  6  working  days 
1  0-5  [$150K  salary)  with  1  working  day 

1  0-5  [$150K  salary)  with  2  working  days 
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14.  ABSTRACT 

The  advancement  in  small  unmanned  aerial  vehicle  (SUAV)  technology  has  brought  a  new  revolution  in  the  military  domain. 
Their  uses  have  become  more  synonymous  with  intelligence,  surveillance,  and  reconnaissance  missions.  Concerns  over  their 
flight  test  safety  and  accountability  have  been  addressed  in  multiple  policies  to  mitigate  mishaps  and  increase  proper 
jaccountability.  However,  current  DoD  and  FAA  mandated  regulations  and  policies  concerning  UAV  flight  tests  are 
sometimes  considered  slow  and  time-consuming,  which  may  lead  to  delays  in  UAV  research  and  development.  This  study 
jexplores  the  quantitative  and  qualitative  measure  of  benefits  associated  with  an  abbreviated  flight  test  process  for  SUAVs. 
Specifically,  it  examines  the  current  agreement  between  two  major  USAF  research  centers  regarding  the  SUAV  flight  test 
approval  process.  This  research  utilized  high-level  multidisciplinary  approaches  and  techniques  including  qualitative  cost- 
benefit  analysis,  interviews,  value  stream  mapping  (VSM)  analysis,  and  heuristic  risk  analysis  to  evaluate  the  current-state 
process.  The  findings  conclude  that  there  is  a  slight  economic  cost  and  schedule  savings  in  an  abbreviated  process. 
Additionally,  this  research  finds  no  correlation  between  SUAV  flight  mishaps  and  system  maturity.  This  research  proposes 
using  a  streamlined  process  for  additional  safety  reviews  to  eliminate  non-value  added  process  steps  considered  unnecessary 
due  to  the  nature  of  the  SUAV  complexity.  Furthermore,  this  study  recommends  using  a  decision  rule  matrix  based  on  the 
total  cost  of  the  SUAV  and  its  weight  and  energy  at  impact  for  choosing  an  abbreviated  flight  test  safety  review  process. 
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